2021-03-09T09:32:46.822 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-09T09:32:46.824 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/crash} --verbose --owner scheduled_kchai@teuthology --name kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi --archive /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949129 -- /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949129/orig.config.yaml 2021-03-09T09:32:46.864 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-09T09:32:46.920 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949129 branch: wip-kefu-testing-2021-03-09-1428 description: rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/crash} email: tchaikov@gmail.com first_in_suite: false job_id: '5949129' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 2 size: 30 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-kefu-testing-2021-03-09-1428 ceph: conf: client: debug client: 20 debug mgrc: 20 debug ms: 1 global: mon election default strategy: 1 mds: debug mgrc: 20 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug mgrc: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2021-03-09-1428 sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mgr.x - mon.a - mon.c - mds.a - mds.c - osd.0 - client.0 - - mgr.y - mgr.z - mon.b - mds.b - osd.1 - osd.2 - osd.3 - client.1 sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-03-09-1428 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 targets: smithi052.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD0ZQp1ZeElCiOPSkQH7WdORAWptn1TAh5z/J44mMHlsgABBAat/r0NZfbQkgkGygkwfwvY8RMvHvDXG5ldpAeIN7vRK0hoVx1iHY1rNA+E1CEV+vMSK43oXdiARX59vGm0X4Y5qgADJsFHMzzbvzXViev3LXiZwvgFig3PBstn/k4OY7kd97wqaM8bhhwKtf0d+lJw6o+s0rco7oThxj/Cw+7R2OKFRWnStGn2u8d1KtXtVGk16nNXDDLbr3XAyppKCkhIATtGcKKoEQlFDYgRK2Rp2pxKuzeFQ4gXhugxWKKAVh2DiIxNuWJworpwesE/cAZ2w/U5kJJgr1ELHlsv smithi194.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCv/RHVZrPckeonnNJlHjUDhH9eL2nn5+KFKF90W7xHY3W4jTznBIlKDGtb/LfK+Sv0CBi8T0g9mLJ618uiFS6qQFqtCS45ibuedVAhW+thV530gzC4DVWok85idX2rrG7mvafE32HsqioMpoaZ2zvYHzDaMRJ5X6beBNNthT4qL1MuzPwP6giVCp5tyQyae1T9LfykU4ZDZj2wUPBhFMwvip5MO3iAhFoz7kxEVBBx+KVFZ76sQVHZ74wEBaZOcthg6SQQYlbXVQAME2eC4+/JrdWM4YfIwjbGPdbjfTSvY/aEhzpkuvzlviaKyzkzfHxRUyw9DZC/nhDsPe5NlVTj tasks: - install: null - ceph: log-ignorelist: - overall HEALTH_ - \(MGR_DOWN\) - \(PG_ - \(RECENT_CRASH\) - replacing it with standby - No standby daemons available wait-for-scrub: false - cephfs_test_runner: modules: - tasks.mgr.test_crash teuthology_branch: master teuthology_sha1: 3082387bba74fcd24c9700593d10418152d53c97 timestamp: 2021-03-09_08:22:40 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.25973 2021-03-09T09:32:46.920 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa; will attempt to use it 2021-03-09T09:32:46.927 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa/tasks 2021-03-09T09:32:46.928 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-09T09:32:46.934 INFO:teuthology.task.internal:Checking packages... 2021-03-09T09:32:47.001 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '89fb622b10ca2e58fe0c913a8956377a886e2ab4' 2021-03-09T09:32:47.001 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-09T09:32:47.002 INFO:teuthology.packaging:ref: None 2021-03-09T09:32:47.002 INFO:teuthology.packaging:tag: None 2021-03-09T09:32:47.003 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-03-09-1428 2021-03-09T09:32:47.003 INFO:teuthology.packaging:sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T09:32:47.004 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-kefu-testing-2021-03-09-1428 2021-03-09T09:32:47.185 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:32:47.186 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-09T09:32:47.202 INFO:teuthology.task.internal:no buildpackages task found 2021-03-09T09:32:47.203 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-09T09:32:47.257 INFO:teuthology.task.internal:Saving configuration 2021-03-09T09:32:47.277 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-09T09:32:47.300 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-09T09:32:47.322 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-09 09:27:35.968080', 'description': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949129', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKqushaBPt4ES5vcdLOCyRd6OYUPQIRNkq+XtkjptQ1U', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi052.front.sepia.ceph.com'} 2021-03-09T09:32:47.349 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-09 09:27:35.968098', 'description': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949129', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPjh0v1m8cpZ4nkW5JM7hxh5h7M0g0Eyrix65GUuQLlY', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi194.front.sepia.ceph.com'} 2021-03-09T09:32:47.351 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-09T09:32:47.371 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2021-03-09T09:32:47.371 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2021-03-09T09:32:47.372 INFO:teuthology.run_tasks:Running task console_log... 2021-03-09T09:32:47.518 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb42f26f9d8>, signals=[15]) 2021-03-09T09:32:47.519 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-09T09:32:47.530 INFO:teuthology.task.internal:Opening connections... 2021-03-09T09:32:47.530 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2021-03-09T09:32:47.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T09:32:47.620 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2021-03-09T09:32:47.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T09:32:47.745 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-09T09:32:47.760 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2021-03-09T09:32:47.782 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2021-03-09T09:32:47.784 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2021-03-09T09:32:47.834 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="Ubuntu" 2021-03-09T09:32:47.835 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-09T09:32:47.835 INFO:teuthology.orchestra.run.smithi052.stdout:ID=ubuntu 2021-03-09T09:32:47.835 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE=debian 2021-03-09T09:32:47.836 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-09T09:32:47.836 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="18.04" 2021-03-09T09:32:47.837 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-09T09:32:47.837 INFO:teuthology.orchestra.run.smithi052.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-09T09:32:47.838 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-09T09:32:47.839 INFO:teuthology.orchestra.run.smithi052.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-09T09:32:47.840 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_CODENAME=bionic 2021-03-09T09:32:47.840 INFO:teuthology.orchestra.run.smithi052.stdout:UBUNTU_CODENAME=bionic 2021-03-09T09:32:47.841 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2021-03-09T09:32:47.865 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2021-03-09T09:32:47.909 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2021-03-09T09:32:47.910 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2021-03-09T09:32:47.961 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="Ubuntu" 2021-03-09T09:32:47.961 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-09T09:32:47.961 INFO:teuthology.orchestra.run.smithi194.stdout:ID=ubuntu 2021-03-09T09:32:47.962 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE=debian 2021-03-09T09:32:47.962 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-09T09:32:47.962 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="18.04" 2021-03-09T09:32:47.962 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-09T09:32:47.963 INFO:teuthology.orchestra.run.smithi194.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-09T09:32:47.963 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-09T09:32:47.963 INFO:teuthology.orchestra.run.smithi194.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-09T09:32:47.963 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_CODENAME=bionic 2021-03-09T09:32:47.963 INFO:teuthology.orchestra.run.smithi194.stdout:UBUNTU_CODENAME=bionic 2021-03-09T09:32:47.964 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2021-03-09T09:32:47.979 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-09T09:32:47.991 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-09T09:32:48.110 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-09T09:32:48.111 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2021-03-09T09:32:48.113 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2021-03-09T09:32:48.119 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-09T09:32:48.186 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-09T09:32:48.186 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2021-03-09T09:32:48.189 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2021-03-09T09:32:48.217 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-09T09:32:48.261 INFO:teuthology.run_tasks:Running task kernel... 2021-03-09T09:32:48.318 DEBUG:teuthology.task.kernel:normalized config {'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'mds.c': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mgr.z': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}} 2021-03-09T09:32:48.319 INFO:teuthology.task.kernel:config {'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-09T09:32:48.320 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2021-03-09T09:32:48.326 INFO:teuthology.orchestra.run.smithi052.stdout:4.15.0-136-generic 2021-03-09T09:32:48.326 INFO:teuthology.task.kernel:Running kernel on smithi052: 4.15.0-136-generic 2021-03-09T09:32:48.326 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get clean 2021-03-09T09:32:48.432 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get update 2021-03-09T09:32:48.725 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T09:32:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-09T09:32:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-09T09:32:49.041 INFO:teuthology.orchestra.run.smithi052.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-09T09:32:49.175 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [926 kB] 2021-03-09T09:32:49.316 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,913 kB] 2021-03-09T09:32:49.566 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,573 kB] 2021-03-09T09:32:49.677 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [302 kB] 2021-03-09T09:32:49.693 INFO:teuthology.orchestra.run.smithi052.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [245 kB] 2021-03-09T09:32:49.702 INFO:teuthology.orchestra.run.smithi052.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-03-09T09:32:49.702 INFO:teuthology.orchestra.run.smithi052.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [31.7 kB] 2021-03-09T09:32:49.703 INFO:teuthology.orchestra.run.smithi052.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [979 kB] 2021-03-09T09:32:49.735 INFO:teuthology.orchestra.run.smithi052.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,228 kB] 2021-03-09T09:32:49.746 INFO:teuthology.orchestra.run.smithi052.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,113 kB] 2021-03-09T09:32:49.770 INFO:teuthology.orchestra.run.smithi052.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [266 kB] 2021-03-09T09:32:49.776 INFO:teuthology.orchestra.run.smithi052.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [249 kB] 2021-03-09T09:32:49.778 INFO:teuthology.orchestra.run.smithi052.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-03-09T09:32:49.779 INFO:teuthology.orchestra.run.smithi052.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [35.3 kB] 2021-03-09T09:32:49.779 INFO:teuthology.orchestra.run.smithi052.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,561 kB] 2021-03-09T09:32:49.782 INFO:teuthology.orchestra.run.smithi052.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-03-09T09:32:49.783 INFO:teuthology.orchestra.run.smithi052.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-03-09T09:32:49.783 INFO:teuthology.orchestra.run.smithi052.stdout:Get:22 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-03-09T09:32:49.846 INFO:teuthology.orchestra.run.smithi052.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,719 kB] 2021-03-09T09:32:49.890 INFO:teuthology.orchestra.run.smithi052.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-03-09T09:32:49.891 INFO:teuthology.orchestra.run.smithi052.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-03-09T09:32:49.891 INFO:teuthology.orchestra.run.smithi052.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-03-09T09:32:51.266 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 12.5 MB in 3s (4,815 kB/s) 2021-03-09T09:32:52.781 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-09T09:32:52.831 DEBUG:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-09T09:32:52.911 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-09T09:32:53.150 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2021-03-09T09:32:53.151 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2021-03-09T09:32:53.236 INFO:teuthology.orchestra.run.smithi052.stderr:E: Unable to locate package linux-image-current-generic 2021-03-09T09:32:53.237 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-09T09:32:53.239 DEBUG:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-09T09:32:53.323 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-09T09:32:53.566 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2021-03-09T09:32:53.567 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2021-03-09T09:32:53.783 INFO:teuthology.orchestra.run.smithi052.stdout:linux-image-generic is already the newest version (4.15.0.136.123). 2021-03-09T09:32:53.783 INFO:teuthology.orchestra.run.smithi052.stdout:linux-image-generic set to manually installed. 2021-03-09T09:32:53.784 INFO:teuthology.orchestra.run.smithi052.stdout:0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T09:32:53.785 DEBUG:teuthology.orchestra.run.smithi052:> dpkg -s linux-image-generic 2021-03-09T09:32:53.801 INFO:teuthology.orchestra.run.smithi052.stdout:Package: linux-image-generic 2021-03-09T09:32:53.801 INFO:teuthology.orchestra.run.smithi052.stdout:Status: install ok installed 2021-03-09T09:32:53.801 INFO:teuthology.orchestra.run.smithi052.stdout:Priority: optional 2021-03-09T09:32:53.801 INFO:teuthology.orchestra.run.smithi052.stdout:Section: kernel 2021-03-09T09:32:53.802 INFO:teuthology.orchestra.run.smithi052.stdout:Installed-Size: 17 2021-03-09T09:32:53.802 INFO:teuthology.orchestra.run.smithi052.stdout:Maintainer: Ubuntu Kernel Team 2021-03-09T09:32:53.802 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: amd64 2021-03-09T09:32:53.802 INFO:teuthology.orchestra.run.smithi052.stdout:Source: linux-meta 2021-03-09T09:32:53.803 INFO:teuthology.orchestra.run.smithi052.stdout:Version: 4.15.0.136.123 2021-03-09T09:32:53.803 INFO:teuthology.orchestra.run.smithi052.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-09T09:32:53.803 INFO:teuthology.orchestra.run.smithi052.stdout:Depends: linux-image-4.15.0-136-generic, linux-modules-extra-4.15.0-136-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-09T09:32:53.803 INFO:teuthology.orchestra.run.smithi052.stdout:Recommends: thermald 2021-03-09T09:32:53.804 INFO:teuthology.orchestra.run.smithi052.stdout:Description: Generic Linux kernel image 2021-03-09T09:32:53.804 INFO:teuthology.orchestra.run.smithi052.stdout: This package will always depend on the latest generic kernel image 2021-03-09T09:32:53.804 INFO:teuthology.orchestra.run.smithi052.stdout: available. 2021-03-09T09:32:53.804 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-09T09:32:53.805 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2021-03-09T09:32:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:4.15.0-136-generic 2021-03-09T09:32:53.811 INFO:teuthology.task.kernel:Running kernel on smithi194: 4.15.0-136-generic 2021-03-09T09:32:53.811 DEBUG:teuthology.orchestra.run.smithi194:> sudo apt-get clean 2021-03-09T09:32:54.020 DEBUG:teuthology.orchestra.run.smithi194:> sudo apt-get update 2021-03-09T09:32:54.283 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T09:32:54.284 INFO:teuthology.orchestra.run.smithi194.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-09T09:32:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-09T09:32:54.595 INFO:teuthology.orchestra.run.smithi194.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-09T09:32:54.655 INFO:teuthology.orchestra.run.smithi194.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [926 kB] 2021-03-09T09:32:54.789 INFO:teuthology.orchestra.run.smithi194.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,228 kB] 2021-03-09T09:32:55.025 INFO:teuthology.orchestra.run.smithi194.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,573 kB] 2021-03-09T09:32:55.123 INFO:teuthology.orchestra.run.smithi194.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,913 kB] 2021-03-09T09:32:55.143 INFO:teuthology.orchestra.run.smithi194.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [302 kB] 2021-03-09T09:32:55.158 INFO:teuthology.orchestra.run.smithi194.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-03-09T09:32:55.159 INFO:teuthology.orchestra.run.smithi194.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [245 kB] 2021-03-09T09:32:55.164 INFO:teuthology.orchestra.run.smithi194.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [31.7 kB] 2021-03-09T09:32:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [979 kB] 2021-03-09T09:32:55.233 INFO:teuthology.orchestra.run.smithi194.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,113 kB] 2021-03-09T09:32:55.255 INFO:teuthology.orchestra.run.smithi194.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-03-09T09:32:55.255 INFO:teuthology.orchestra.run.smithi194.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [266 kB] 2021-03-09T09:32:55.257 INFO:teuthology.orchestra.run.smithi194.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [249 kB] 2021-03-09T09:32:55.260 INFO:teuthology.orchestra.run.smithi194.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-03-09T09:32:55.260 INFO:teuthology.orchestra.run.smithi194.stdout:Get:19 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-03-09T09:32:55.261 INFO:teuthology.orchestra.run.smithi194.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-03-09T09:32:55.261 INFO:teuthology.orchestra.run.smithi194.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [35.3 kB] 2021-03-09T09:32:55.262 INFO:teuthology.orchestra.run.smithi194.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,719 kB] 2021-03-09T09:32:55.318 INFO:teuthology.orchestra.run.smithi194.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,561 kB] 2021-03-09T09:32:55.345 INFO:teuthology.orchestra.run.smithi194.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-03-09T09:32:55.345 INFO:teuthology.orchestra.run.smithi194.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-03-09T09:32:55.345 INFO:teuthology.orchestra.run.smithi194.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-03-09T09:32:56.605 INFO:teuthology.orchestra.run.smithi194.stdout:Fetched 12.5 MB in 2s (5,323 kB/s) 2021-03-09T09:32:58.055 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2021-03-09T09:32:58.102 DEBUG:teuthology.orchestra.run.smithi194:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-09T09:32:58.182 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2021-03-09T09:32:58.422 INFO:teuthology.orchestra.run.smithi194.stdout:Building dependency tree... 2021-03-09T09:32:58.422 INFO:teuthology.orchestra.run.smithi194.stdout:Reading state information... 2021-03-09T09:32:58.502 INFO:teuthology.orchestra.run.smithi194.stderr:E: Unable to locate package linux-image-current-generic 2021-03-09T09:32:58.503 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-09T09:32:58.504 DEBUG:teuthology.orchestra.run.smithi194:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-09T09:32:58.583 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2021-03-09T09:32:58.814 INFO:teuthology.orchestra.run.smithi194.stdout:Building dependency tree... 2021-03-09T09:32:58.815 INFO:teuthology.orchestra.run.smithi194.stdout:Reading state information... 2021-03-09T09:32:59.023 INFO:teuthology.orchestra.run.smithi194.stdout:linux-image-generic is already the newest version (4.15.0.136.123). 2021-03-09T09:32:59.023 INFO:teuthology.orchestra.run.smithi194.stdout:linux-image-generic set to manually installed. 2021-03-09T09:32:59.023 INFO:teuthology.orchestra.run.smithi194.stdout:0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T09:32:59.024 DEBUG:teuthology.orchestra.run.smithi194:> dpkg -s linux-image-generic 2021-03-09T09:32:59.044 INFO:teuthology.orchestra.run.smithi194.stdout:Package: linux-image-generic 2021-03-09T09:32:59.044 INFO:teuthology.orchestra.run.smithi194.stdout:Status: install ok installed 2021-03-09T09:32:59.044 INFO:teuthology.orchestra.run.smithi194.stdout:Priority: optional 2021-03-09T09:32:59.045 INFO:teuthology.orchestra.run.smithi194.stdout:Section: kernel 2021-03-09T09:32:59.045 INFO:teuthology.orchestra.run.smithi194.stdout:Installed-Size: 17 2021-03-09T09:32:59.045 INFO:teuthology.orchestra.run.smithi194.stdout:Maintainer: Ubuntu Kernel Team 2021-03-09T09:32:59.045 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: amd64 2021-03-09T09:32:59.045 INFO:teuthology.orchestra.run.smithi194.stdout:Source: linux-meta 2021-03-09T09:32:59.046 INFO:teuthology.orchestra.run.smithi194.stdout:Version: 4.15.0.136.123 2021-03-09T09:32:59.046 INFO:teuthology.orchestra.run.smithi194.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-09T09:32:59.046 INFO:teuthology.orchestra.run.smithi194.stdout:Depends: linux-image-4.15.0-136-generic, linux-modules-extra-4.15.0-136-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-09T09:32:59.047 INFO:teuthology.orchestra.run.smithi194.stdout:Recommends: thermald 2021-03-09T09:32:59.047 INFO:teuthology.orchestra.run.smithi194.stdout:Description: Generic Linux kernel image 2021-03-09T09:32:59.047 INFO:teuthology.orchestra.run.smithi194.stdout: This package will always depend on the latest generic kernel image 2021-03-09T09:32:59.047 INFO:teuthology.orchestra.run.smithi194.stdout: available. 2021-03-09T09:32:59.048 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-09T09:32:59.048 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2021-03-09T09:32:59.048 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-09T09:32:59.067 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2021-03-09T09:32:59.096 INFO:teuthology.task.kernel:Enabling kdb on mgr.y... 2021-03-09T09:32:59.096 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-09T09:32:59.118 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2021-03-09T09:32:59.131 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-09T09:32:59.141 INFO:teuthology.task.internal:Creating test directory... 2021-03-09T09:32:59.143 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-09T09:32:59.146 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-09T09:32:59.179 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-09T09:32:59.189 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-09T09:32:59.214 INFO:teuthology.task.internal:Creating archive directory... 2021-03-09T09:32:59.214 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-09T09:32:59.217 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-09T09:32:59.250 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-09T09:32:59.263 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-09T09:32:59.264 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-09T09:32:59.291 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-09T09:32:59.313 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T09:32:59.314 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T09:32:59.322 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T09:32:59.322 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T09:32:59.324 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-09T09:32:59.336 INFO:teuthology.task.internal:Configuring sudo... 2021-03-09T09:32:59.336 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-09T09:32:59.367 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-09T09:32:59.382 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-09T09:32:59.399 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-09T09:32:59.400 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-09T09:32:59.423 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-09T09:32:59.430 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T09:32:59.475 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T09:32:59.523 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:32:59.524 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-09T09:32:59.586 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T09:32:59.593 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T09:32:59.643 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:32:59.644 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-09T09:32:59.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2021-03-09T09:32:59.721 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2021-03-09T09:32:59.800 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-09T09:32:59.812 INFO:teuthology.task.internal:Starting timer... 2021-03-09T09:32:59.812 INFO:teuthology.run_tasks:Running task pcp... 2021-03-09T09:32:59.845 INFO:teuthology.run_tasks:Running task selinux... 2021-03-09T09:32:59.868 DEBUG:teuthology.task.selinux:Excluding smithi052: OS 'ubuntu' does not support SELinux 2021-03-09T09:32:59.868 DEBUG:teuthology.task.selinux:Excluding smithi194: OS 'ubuntu' does not support SELinux 2021-03-09T09:32:59.869 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-09T09:32:59.869 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-03-09T09:32:59.869 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-09T09:32:59.869 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-09T09:32:59.896 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-03-09T09:32:59.919 INFO:teuthology.repo_utils:Fetching master from origin 2021-03-09T09:33:00.116 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-09T09:33:00.343 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-09T09:33:00.354 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-09T09:33:01.135 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-09T09:33:02.966 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-09T09:33:05.344 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-09T09:33:05.407 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:05 +0000 (0:00:00.091) 0:00:00.091 ********* 2021-03-09T09:33:05.426 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:05 +0000 (0:00:00.018) 0:00:00.110 ********* 2021-03-09T09:33:07.191 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-09T09:33:07.193 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi052.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi052.front.sepia.ceph.com] 2021-03-09T09:33:07.260 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:07 +0000 (0:00:01.834) 0:00:01.945 ********* 2021-03-09T09:33:07.291 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi194.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi194.front.sepia.ceph.com] 2021-03-09T09:33:07.343 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:07 +0000 (0:00:00.082) 0:00:02.027 ********* 2021-03-09T09:33:07.526 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-03-09T09:33:07.528 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T09:33:07.551 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:07 +0000 (0:00:00.207) 0:00:02.235 ********* 2021-03-09T09:33:07.571 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:07 +0000 (0:00:00.020) 0:00:02.255 ********* 2021-03-09T09:33:08.250 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-03-09T09:33:08.252 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-09T09:33:08.274 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:08 +0000 (0:00:00.702) 0:00:02.958 ********* 2021-03-09T09:33:08.280 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-09T09:33:08.305 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:08 +0000 (0:00:00.031) 0:00:02.989 ********* 2021-03-09T09:33:08.641 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-03-09T09:33:08.643 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.066924", "end": "2021-03-09 09:33:08.619810", "rc": 0, "start": "2021-03-09 09:33:08.552886", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-09T09:33:08.664 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:08 +0000 (0:00:00.358) 0:00:03.348 ********* 2021-03-09T09:33:08.670 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.083562", "end": "2021-03-09 09:33:08.594962", "rc": 0, "start": "2021-03-09 09:33:08.511400", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-09T09:33:08.696 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:08 +0000 (0:00:00.032) 0:00:03.381 ********* 2021-03-09T09:33:08.933 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-03-09T09:33:08.935 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi194.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:33:08.959 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:08 +0000 (0:00:00.262) 0:00:03.643 ********* 2021-03-09T09:33:08.979 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:08 +0000 (0:00:00.019) 0:00:03.663 ********* 2021-03-09T09:33:09.801 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-03-09T09:33:09.803 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282388.9921548-6276-112089421625566/source", "state": "file", "uid": 0} 2021-03-09T09:33:09.838 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:09 +0000 (0:00:00.858) 0:00:04.521 ********* 2021-03-09T09:33:10.321 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282389.0123034-6278-54041936852601/source", "state": "file", "uid": 0} Tuesday 09 March 2021 09:33:09 +0000 (0:00:00.040) 0:00:04.562 ********* TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJH 2021-03-09T09:33:10.322 INFO:teuthology.task.ansible.out:Fm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-09T09:33:10.334 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-09T09:33:10.336 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-09T09:33:10.410 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-03-09T09:33:10.413 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-03-09T09:33:10.559 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:10 +0000 (0:00:00.680) 0:00:05.243 ********* 2021-03-09T09:33:10.579 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:10 +0000 (0:00:00.019) 0:00:05.263 ********* 2021-03-09T09:33:11.311 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi052.front.sepia.ceph.com] 2021-03-09T09:33:11.321 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] 2021-03-09T09:33:11.385 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-03-09T09:33:11.387 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:11 +0000 (0:00:00.806) 0:00:06.069 ********* 2021-03-09T09:33:11.412 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:11 +0000 (0:00:00.026) 0:00:06.096 ********* 2021-03-09T09:33:12.195 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-09T09:33:12.197 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.430696", "end": "2021-03-09 09:33:12.173731", "rc": 0, "start": "2021-03-09 09:33:11.743035", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.430397", "end": "2021-03-09 09:33:12.173772", "rc": 0, "start": "2021-03-09 09:33:11.743375", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-09T09:33:12.222 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.810) 0:00:06.906 ********* 2021-03-09T09:33:12.245 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.022) 0:00:06.929 ********* 2021-03-09T09:33:12.354 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-09T09:33:12.356 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009176", "end": "2021-03-09 09:33:12.339472", "failed_when_result": false, "rc": 0, "start": "2021-03-09 09:33:12.330296", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-09T09:33:12.384 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.139) 0:00:07.068 ********* 2021-03-09T09:33:12.402 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.012608", "end": "2021-03-09 09:33:12.331651", "failed_when_result": false, "rc": 0, "start": "2021-03-09 09:33:12.319043", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-09T09:33:12.426 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.041) 0:00:07.110 ********* 2021-03-09T09:33:12.453 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-09T09:33:12.455 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:12.477 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.051) 0:00:07.162 ********* 2021-03-09T09:33:12.494 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:12.518 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.040) 0:00:07.203 ********* 2021-03-09T09:33:12.545 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-03-09T09:33:12.547 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:12.569 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.050) 0:00:07.253 ********* 2021-03-09T09:33:12.586 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:12.610 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.040) 0:00:07.294 ********* 2021-03-09T09:33:12.636 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-03-09T09:33:12.638 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:12.660 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.050) 0:00:07.345 ********* 2021-03-09T09:33:12.687 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:12.711 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.050) 0:00:07.395 ********* 2021-03-09T09:33:12.727 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-09T09:33:12.728 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:12.751 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.040) 0:00:07.435 ********* 2021-03-09T09:33:12.779 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:12.802 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.051) 0:00:07.486 ********* 2021-03-09T09:33:12.829 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-09T09:33:12.831 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:12.854 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.051) 0:00:07.538 ********* 2021-03-09T09:33:12.881 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:12.908 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:12 +0000 (0:00:00.054) 0:00:07.592 ********* 2021-03-09T09:33:13.060 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-03-09T09:33:13.062 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:33:13.084 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:13 +0000 (0:00:00.175) 0:00:07.768 ********* 2021-03-09T09:33:13.101 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:33:13.125 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:13 +0000 (0:00:00.041) 0:00:07.809 ********* 2021-03-09T09:33:13.328 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T09:33:13.330 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-09T09:33:13.361 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-09T09:33:13.476 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-09T09:33:13.498 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-09T09:33:13.613 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-09T09:33:13.646 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-09T09:33:13.750 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-09T09:33:13.783 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-09T09:33:13.897 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-09T09:33:13.929 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-09T09:33:14.044 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-09T09:33:14.056 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-09T09:33:14.191 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-09T09:33:14.203 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-09T09:33:14.338 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-09T09:33:14.351 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-09T09:33:14.510 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-09T09:33:14.511 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-09T09:33:14.662 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-09T09:33:14.673 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-09T09:33:14.812 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-09T09:33:14.834 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-09T09:33:14.953 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-09T09:33:14.985 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-09T09:33:15.110 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-09T09:33:15.132 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-09T09:33:15.247 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-09T09:33:15.280 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-09T09:33:15.405 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-09T09:33:15.427 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-09T09:33:15.552 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-09T09:33:15.574 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-09T09:33:15.700 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-09T09:33:15.722 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-09T09:33:15.857 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-09T09:33:15.868 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-09T09:33:16.014 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-09T09:33:16.016 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-09T09:33:16.161 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-09T09:33:16.162 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-09T09:33:16.298 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-09T09:33:16.309 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-09T09:33:16.445 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-09T09:33:16.446 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-09T09:33:16.582 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-09T09:33:16.593 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-09T09:33:16.729 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-09T09:33:16.730 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-09T09:33:16.866 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-09T09:33:16.872 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-09T09:33:17.013 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-09T09:33:17.015 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-09T09:33:17.151 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-09T09:33:17.162 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-09T09:33:17.298 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-09T09:33:17.309 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-09T09:33:17.445 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-09T09:33:17.446 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-09T09:33:17.582 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-09T09:33:17.593 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-09T09:33:17.729 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-09T09:33:17.740 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-09T09:33:17.886 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-09T09:33:17.887 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-09T09:33:18.023 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-09T09:33:18.024 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-09T09:33:18.170 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-09T09:33:18.171 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-09T09:33:18.317 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-09T09:33:18.318 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-09T09:33:18.464 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-09T09:33:18.465 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-09T09:33:18.601 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-09T09:33:18.612 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-09T09:33:18.748 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-09T09:33:18.759 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-09T09:33:18.888 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T09:33:18.900 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T09:33:19.035 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-09T09:33:19.047 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-09T09:33:19.182 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-09T09:33:19.194 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-09T09:33:19.319 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-09T09:33:19.341 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-09T09:33:19.465 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-09T09:33:19.477 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-09T09:33:19.613 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-09T09:33:19.624 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-09T09:33:19.750 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-09T09:33:19.772 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-09T09:33:19.897 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-09T09:33:19.919 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-09T09:33:20.044 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T09:33:20.055 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T09:33:20.191 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-09T09:33:20.202 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-09T09:33:20.327 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-09T09:33:20.338 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-09T09:33:20.485 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-09T09:33:20.486 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-09T09:33:20.622 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-09T09:33:20.623 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-09T09:33:20.768 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-09T09:33:20.780 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-09T09:33:20.915 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-09T09:33:20.917 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-09T09:33:21.052 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-09T09:33:21.064 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-09T09:33:21.200 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-09T09:33:21.211 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-09T09:33:21.340 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-09T09:33:21.351 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-09T09:33:21.487 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-09T09:33:21.498 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-09T09:33:21.624 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-09T09:33:21.646 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-09T09:33:21.762 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-09T09:33:21.784 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-09T09:33:21.910 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-09T09:33:21.933 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-09T09:33:22.047 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-09T09:33:22.079 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-09T09:33:22.194 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-09T09:33:22.226 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-09T09:33:22.352 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-09T09:33:22.363 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-09T09:33:22.500 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T09:33:22.512 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T09:33:22.648 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-09T09:33:22.659 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-09T09:33:22.794 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-09T09:33:22.806 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-09T09:33:22.931 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-09T09:33:22.942 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-09T09:33:23.078 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-09T09:33:23.080 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-09T09:33:23.215 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-09T09:33:23.226 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-09T09:33:23.363 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-09T09:33:23.364 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-09T09:33:23.500 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-09T09:33:23.511 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-09T09:33:23.637 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-09T09:33:23.659 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-09T09:33:23.775 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-09T09:33:23.797 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-09T09:33:23.921 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-09T09:33:23.954 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-09T09:33:24.069 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-09T09:33:24.091 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-09T09:33:24.217 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-09T09:33:24.228 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-09T09:33:24.353 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-09T09:33:24.375 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-09T09:33:24.491 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-09T09:33:24.513 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-09T09:33:24.649 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-09T09:33:24.775 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-09T09:33:24.797 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-09T09:33:24.911 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-09T09:33:24.943 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-09T09:33:25.058 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-09T09:33:25.090 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-09T09:33:25.195 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-09T09:33:25.237 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-09T09:33:25.342 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-09T09:33:25.385 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-09T09:33:25.489 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-09T09:33:25.521 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-09T09:33:25.636 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-09T09:33:25.668 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-09T09:33:25.772 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-09T09:33:25.815 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-09T09:33:25.919 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-09T09:33:25.952 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-09T09:33:26.068 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-09T09:33:26.100 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-09T09:33:26.204 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-09T09:33:26.247 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-09T09:33:26.351 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-09T09:33:26.383 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-09T09:33:26.488 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-09T09:33:26.530 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-09T09:33:26.635 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-09T09:33:26.677 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-09T09:33:26.771 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-09T09:33:26.824 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-09T09:33:26.909 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-09T09:33:26.962 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-09T09:33:27.056 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-09T09:33:27.110 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-09T09:33:27.194 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-09T09:33:27.257 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-09T09:33:27.341 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-09T09:33:27.395 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-09T09:33:27.479 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-09T09:33:27.543 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-09T09:33:27.616 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-09T09:33:27.679 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-09T09:33:27.752 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-09T09:33:27.826 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-09T09:33:27.899 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-09T09:33:27.973 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T09:33:28.046 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T09:33:28.109 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-09T09:33:28.193 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-09T09:33:28.256 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-09T09:33:28.329 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-09T09:33:28.403 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-09T09:33:28.477 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-09T09:33:28.551 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-09T09:33:28.624 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-09T09:33:28.698 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-09T09:33:28.771 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-09T09:33:28.856 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-09T09:33:28.909 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-09T09:33:28.993 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-09T09:33:29.056 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-09T09:33:29.140 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-09T09:33:29.204 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-09T09:33:29.278 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-09T09:33:29.352 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-09T09:33:29.426 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-09T09:33:29.490 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-09T09:33:29.574 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-09T09:33:29.639 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-09T09:33:29.713 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-09T09:33:29.787 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-09T09:33:29.861 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-09T09:33:29.924 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-09T09:33:30.008 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-09T09:33:30.072 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-09T09:33:30.157 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-09T09:33:30.221 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-09T09:33:30.306 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-09T09:33:30.359 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-09T09:33:30.454 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-09T09:33:30.507 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-09T09:33:30.601 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-09T09:33:30.654 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-09T09:33:30.738 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-09T09:33:30.791 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-09T09:33:30.886 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-09T09:33:30.939 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-09T09:33:31.033 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-09T09:33:31.076 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-09T09:33:31.181 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-09T09:33:31.183 INFO:teuthology.task.ansible.out: 2021-03-09T09:33:31.225 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-09T09:33:31.227 INFO:teuthology.task.ansible.out: 2021-03-09T09:33:31.330 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-09T09:33:31.364 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-09T09:33:31.468 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-09T09:33:31.500 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-09T09:33:31.615 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-09T09:33:31.648 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-09T09:33:31.763 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-09T09:33:31.785 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-09T09:33:31.921 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-09T09:33:31.933 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-09T09:33:32.058 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-09T09:33:32.070 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-09T09:33:32.217 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-09T09:33:32.342 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-09T09:33:32.364 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-09T09:33:32.489 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-09T09:33:32.511 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-09T09:33:32.637 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-09T09:33:32.648 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-09T09:33:32.784 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-09T09:33:32.796 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-09T09:33:32.921 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-09T09:33:32.943 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-09T09:33:33.064 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-09T09:33:33.086 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-09T09:33:33.213 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-09T09:33:33.235 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-09T09:33:33.350 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-09T09:33:33.382 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-09T09:33:33.606 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:33 +0000 (0:00:20.481) 0:00:28.290 ********* 2021-03-09T09:33:33.613 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-09T09:33:33.713 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:33 +0000 (0:00:00.106) 0:00:28.397 ********* 2021-03-09T09:33:33.719 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-09T09:33:33.747 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:33 +0000 (0:00:00.033) 0:00:28.431 ********* 2021-03-09T09:33:33.803 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:33 +0000 (0:00:00.056) 0:00:28.487 ********* 2021-03-09T09:33:33.842 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-09T09:33:33.844 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-09T09:33:33.846 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-09T09:33:33.848 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-09T09:33:33.849 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-09T09:33:33.851 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5 2021-03-09T09:33:33.851 INFO:teuthology.task.ansible.out:mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-09T09:33:33.853 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-09T09:33:33.854 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-09T09:33:33.856 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-09T09:33:33.858 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-09T09:33:33.859 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-09T09:33:33.861 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-09T09:33:33.862 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-noteboo 2021-03-09T09:33:33.863 INFO:teuthology.task.ansible.out:k LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-09T09:33:33.865 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-09T09:33:33.892 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:33 +0000 (0:00:00.088) 0:00:28.576 ********* 2021-03-09T09:33:33.902 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-09T09:33:33.904 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-09T09:33:33.905 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-09T09:33:33.907 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-09T09:33:33.908 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name 2021-03-09T09:33:33.909 INFO:teuthology.task.ansible.out:": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-09T09:33:33.910 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-09T09:33:33.912 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-09T09:33:33.914 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-09T09:33:33.915 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-09T09:33:33.917 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-09T09:33:33.919 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-09T09:33:33.921 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-09T09:33:33.923 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-09T09:33:33.944 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:33 +0000 (0:00:00.052) 0:00:28.628 ********* 2021-03-09T09:33:35.268 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-09T09:33:35.270 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-09T09:33:35.292 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:35 +0000 (0:00:01.347) 0:00:29.976 ********* 2021-03-09T09:33:35.299 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-09T09:33:35.325 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:35 +0000 (0:00:00.033) 0:00:30.009 ********* 2021-03-09T09:33:35.538 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:33:35.539 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-09T09:33:35.551 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-09T09:33:35.677 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-09T09:33:35.688 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-09T09:33:35.813 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-09T09:33:35.814 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-09T09:33:35.961 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-09T09:33:35.963 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1 2021-03-09T09:33:35.965 INFO:teuthology.task.ansible.out:yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-09T09:33:36.097 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-09T09:33:36.098 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-09T09:33:36.233 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-09T09:33:36.234 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7 2021-03-09T09:33:36.236 INFO:teuthology.task.ansible.out:LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-09T09:33:36.364 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-09T09:33:36.365 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-09T09:33:36.499 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-09T09:33:36.501 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-09T09:33:36.636 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-09T09:33:36.647 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:36.649 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1s 2021-03-09T09:33:36.650 INFO:teuthology.task.ansible.out:cr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:36.774 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-09T09:33:36.776 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-09T09:33:36.785 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-09T09:33:36.787 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-09T09:33:36.901 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-09T09:33:36.903 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-09T09:33:36.912 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-09T09:33:36.914 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:36.923 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:36.924 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:36.925 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfm 2021-03-09T09:33:36.926 INFO:teuthology.task.ansible.out:OtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:37.053 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-09T09:33:37.055 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-09T09:33:37.179 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-09T09:33:37.181 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-09T09:33:37.190 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-09T09:33:37.192 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-09T09:33:37.317 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-09T09:33:37.319 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-09T09:33:37.442 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-09T09:33:37.715 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfil 2021-03-09T09:33:37.717 INFO:teuthology.task.ansible.out:e": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-09T09:33:37.721 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-09T09:33:37.731 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-09T09:33:37.857 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-09T09:33:37.877 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-09T09:33:37.995 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-09T09:33:37.997 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-09T09:33:38.017 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-09T09:33:38.019 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-09T09:33:38.132 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-09T09:33:38.135 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-09T09:33:38.153 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-09T09:33:38.156 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-09T09:33:38.268 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-09T09:33:38.290 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-09T09:33:38.405 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-09T09:33:38.426 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-09T09:33:38.531 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-09T09:33:38.532 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-09T09:33:38.562 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-09T09:33:38.564 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-09T09:33:38.657 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-09T09:33:38.699 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-09T09:33:38.782 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-09T09:33:38.834 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-09T09:33:38.918 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-09T09:33:38.961 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-09T09:33:39.045 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-09T09:33:39.087 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-09T09:33:39.182 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-09T09:33:39.224 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-09T09:33:39.308 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-09T09:33:39.310 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-09T09:33:39.361 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-09T09:33:39.363 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-09T09:33:39.434 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-09T09:33:39.486 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-09T09:33:39.571 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-09T09:33:39.623 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-09T09:33:39.707 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-09T09:33:39.749 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-09T09:33:39.843 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-09T09:33:39.895 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-09T09:33:39.969 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-09T09:33:40.022 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-09T09:33:40.096 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-09T09:33:40.158 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-09T09:33:40.221 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T09:33:40.284 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T09:33:40.347 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-09T09:33:40.411 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-09T09:33:40.485 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-09T09:33:40.548 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-09T09:33:40.611 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-09T09:33:40.613 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-09T09:33:40.685 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-09T09:33:40.687 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-09T09:33:40.738 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-09T09:33:40.740 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-09T09:33:40.812 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-09T09:33:40.813 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-09T09:33:40.865 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-09T09:33:40.950 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-09T09:33:40.993 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-09T09:33:41.076 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-09T09:33:41.119 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-09T09:33:41.214 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-09T09:33:41.256 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-09T09:33:41.258 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T09:33:41.341 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-09T09:33:41.342 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T09:33:41.382 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-09T09:33:41.384 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-09T09:33:41.386 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-09T09:33:41.487 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-09T09:33:41.489 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-09T09:33:41.490 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-09T09:33:41.508 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-09T09:33:41.623 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-09T09:33:41.645 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-09T09:33:41.749 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-09T09:33:41.771 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-09T09:33:41.875 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-09T09:33:41.896 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-09T09:33:42.011 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-09T09:33:42.033 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-09T09:33:42.138 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-09T09:33:42.159 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-09T09:33:42.161 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-09T09:33:42.264 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-09T09:33:42.266 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-09T09:33:42.296 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-09T09:33:42.400 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-09T09:33:42.443 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-09T09:33:42.547 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-09T09:33:42.569 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-09T09:33:42.570 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-09T09:33:42.673 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-09T09:33:42.675 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-09T09:33:42.695 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-09T09:33:42.812 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-09T09:33:42.824 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-09T09:33:42.971 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-09T09:33:42.972 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/autho 2021-03-09T09:33:42.974 INFO:teuthology.task.ansible.out:rized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-09T09:33:43.101 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-09T09:33:43.112 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-09T09:33:43.114 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-09T09:33:43.240 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-09T09:33:43.242 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-09T09:33:43.381 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-09T09:33:43.403 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-09T09:33:43.532 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T09:33:43.554 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-09T09:33:43.672 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-09T09:33:43.704 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T09:33:43.800 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-09T09:33:43.858 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-09T09:33:43.937 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-09T09:33:43.979 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-09T09:33:44.062 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-09T09:33:44.126 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-09T09:33:44.199 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-09T09:33:44.262 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-09T09:33:44.335 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-09T09:33:44.398 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-09T09:33:44.461 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-09T09:33:44.524 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-09T09:33:44.587 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-09T09:33:44.660 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-09T09:33:44.713 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-09T09:33:44.786 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-09T09:33:44.839 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-09T09:33:44.923 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-09T09:33:44.965 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-09T09:33:45.049 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-09T09:33:45.095 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-09T09:33:45.178 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-09T09:33:45.231 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-09T09:33:45.314 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-09T09:33:45.346 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-09T09:33:45.440 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-09T09:33:45.472 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-09T09:33:45.566 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-09T09:33:45.598 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-09T09:33:45.702 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-09T09:33:45.723 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-09T09:33:45.725 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-09T09:33:45.828 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-09T09:33:45.849 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-09T09:33:45.964 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-09T09:33:45.966 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-09T09:33:45.975 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-09T09:33:46.090 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-09T09:33:46.101 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-09T09:33:46.216 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-09T09:33:46.218 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-09T09:33:46.352 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-09T09:33:46.354 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-09T09:33:46.355 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-09T09:33:46.477 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-09T09:33:46.478 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-09T09:33:46.479 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-09T09:33:46.614 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-09T09:33:46.616 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-09T09:33:46.740 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-09T09:33:46.741 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-09T09:33:46.743 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-09T09:33:46.865 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-09T09:33:46.867 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-09T09:33:46.991 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-09T09:33:47.002 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-09T09:33:47.128 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-09T09:33:47.129 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-09T09:33:47.257 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-09T09:33:47.258 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-09T09:33:47.385 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-09T09:33:47.406 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-09T09:33:47.522 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-09T09:33:47.524 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-09T09:33:47.543 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-09T09:33:47.648 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-09T09:33:47.680 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-09T09:33:47.784 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-09T09:33:47.816 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-09T09:33:47.819 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-09T09:33:47.911 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-09T09:33:47.943 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-09T09:33:48.047 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-09T09:33:48.079 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-09T09:33:48.175 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-09T09:33:48.207 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-09T09:33:48.301 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-09T09:33:48.343 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-09T09:33:48.437 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-09T09:33:48.439 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T09:33:48.469 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-09T09:33:48.574 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-09T09:33:48.605 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-09T09:33:48.711 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-09T09:33:48.743 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-09T09:33:48.744 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T09:33:48.869 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-09T09:33:48.871 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-09T09:33:48.995 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-09T09:33:48.997 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-09T09:33:49.123 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-09T09:33:49.124 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-09T09:33:49.261 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-09T09:33:49.272 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-09T09:33:49.388 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-09T09:33:49.399 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-09T09:33:49.524 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-09T09:33:49.526 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-09T09:33:49.651 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-09T09:33:49.653 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-09T09:33:49.788 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-09T09:33:49.790 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-09T09:33:49.914 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-09T09:33:49.926 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-09T09:33:50.051 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-09T09:33:50.062 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-09T09:33:50.188 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-09T09:33:50.199 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-09T09:33:50.315 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-09T09:33:50.317 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-09T09:33:50.441 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-09T09:33:50.452 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-09T09:33:50.567 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-09T09:33:50.579 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-09T09:33:50.703 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-09T09:33:50.715 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-09T09:33:50.830 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-09T09:33:50.841 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-09T09:33:50.956 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-09T09:33:50.958 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-09T09:33:50.968 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-09T09:33:51.093 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-09T09:33:51.104 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-09T09:33:51.219 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-09T09:33:51.231 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-09T09:33:51.232 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-09T09:33:51.357 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-09T09:33:51.368 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-09T09:33:51.493 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-09T09:33:51.494 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-09T09:33:51.628 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-09T09:33:51.630 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-09T09:33:51.764 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-09T09:33:51.766 INFO:teuthology.task.ansible.out: ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-09T09:33:51.892 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-09T09:33:51.894 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-09T09:33:52.029 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-09T09:33:52.030 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-09T09:33:52.154 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-09T09:33:52.197 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-09T09:33:52.291 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-09T09:33:52.323 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-09T09:33:52.418 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-09T09:33:52.460 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-09T09:33:52.555 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-09T09:33:52.587 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-09T09:33:52.682 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-09T09:33:52.725 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-09T09:33:52.810 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-09T09:33:52.854 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-09T09:33:52.948 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-09T09:33:52.990 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-09T09:33:53.074 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-09T09:33:53.127 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-09T09:33:53.212 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-09T09:33:53.255 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options" 2021-03-09T09:33:53.257 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-09T09:33:53.339 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-09T09:33:53.393 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-09T09:33:53.501 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:53 +0000 (0:00:18.176) 0:00:48.186 ********* 2021-03-09T09:33:53.510 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options" 2021-03-09T09:33:53.511 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-09T09:33:53.598 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-09T09:33:53.662 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:33:53.663 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ 2021-03-09T09:33:53.665 INFO:teuthology.task.ansible.out:ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcN 2021-03-09T09:33:53.667 INFO:teuthology.task.ansible.out:E/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.689 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:53 +0000 (0:00:00.187) 0:00:48.373 ********* 2021-03-09T09:33:53.697 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.699 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.796 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.807 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.818 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.819 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.829 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-09T09:33:53.830 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.832 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.842 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.844 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.854 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.856 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:53.991 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-09T09:33:53.992 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-09T09:33:53.994 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.004 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.129 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-09T09:33:54.131 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansib 2021-03-09T09:33:54.133 INFO:teuthology.task.ansible.out:le_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.142 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.153 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.163 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.174 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.185 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.196 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.207 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.208 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.218 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.220 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.230 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.241 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.242 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.252 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.263 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-09T09:33:54.264 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.265 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.266 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.277 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.278 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.289 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.290 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.302 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.304 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.314 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.315 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.317 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.327 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.329 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.329 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph 2021-03-09T09:33:54.331 INFO:teuthology.task.ansible.out:.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.340 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.342 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.352 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.354 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.364 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.366 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.376 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.378 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.389 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.390 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.401 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.403 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.413 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.415 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.426 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.426 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.427 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.428 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.438 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.440 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.450 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.451 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.452 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.463 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.464 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.475 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.476 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.478 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.488 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.490 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.501 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.502 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.513 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.515 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.525 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.526 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.537 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.537 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.538 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.538 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.549 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.550 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.561 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.563 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.574 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.575 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.586 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.588 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.599 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.601 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.611 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.613 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.624 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.624 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.625 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.636 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.638 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.648 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.650 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.661 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.661 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.662 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.663 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.673 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.674 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.675 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.686 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.687 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.688 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.690 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.692 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.702 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.703 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.705 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.705 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.717 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.717 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.719 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.719 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.721 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.732 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.732 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.733 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.744 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.746 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbcep 2021-03-09T09:33:54.747 INFO:teuthology.task.ansible.out:h0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.757 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.758 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.760 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.760 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.762 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.772 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.774 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.785 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.786 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.798 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.799 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.800 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.801 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.812 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.813 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.814 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.815 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.815 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.826 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.827 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.827 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.827 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.838 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.839 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.840 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.840 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.851 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.852 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.854 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.855 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.856 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.866 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.868 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.879 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.879 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.881 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.881 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.883 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.894 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.894 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.895 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.895 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.906 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.908 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.919 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.920 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.921 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.933 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.934 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.935 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.936 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.946 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.948 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.959 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.960 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.996 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.998 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:54.999 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.001 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.001 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.003 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.034 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:55 +0000 (0:00:01.345) 0:00:49.718 ********* 2021-03-09T09:33:55.044 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.045 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.047 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.047 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.049 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.062 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.062 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.064 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.076 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.076 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.110 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.143 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:55 +0000 (0:00:00.108) 0:00:49.827 ********* 2021-03-09T09:33:55.164 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:55 +0000 (0:00:00.021) 0:00:49.848 ********* 2021-03-09T09:33:55.213 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:33:55.237 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:33:55 +0000 (0:00:00.073) 0:00:49.921 ********* 2021-03-09T09:33:55.398 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:33:55.400 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-09T09:33:55.462 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-09T09:33:55.535 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-09T09:33:55.608 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-09T09:33:55.671 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-09T09:33:55.754 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-09T09:33:55.807 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-09T09:33:55.893 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-09T09:33:55.945 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-09T09:33:56.029 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-09T09:33:56.082 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-09T09:33:56.166 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-09T09:33:56.230 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-09T09:33:56.321 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-09T09:33:56.356 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-09T09:33:56.460 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-09T09:33:56.503 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-09T09:33:56.505 INFO:teuthology.task.ansible.out: 2021-03-09T09:33:56.598 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-09T09:33:56.640 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-09T09:33:56.734 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-09T09:33:56.776 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-09T09:33:56.880 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-09T09:33:56.912 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-09T09:33:57.016 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-09T09:33:57.048 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-09T09:33:57.153 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-09T09:33:57.185 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-09T09:33:57.300 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-09T09:33:57.321 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-09T09:33:57.436 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-09T09:33:57.458 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-09T09:33:57.573 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-09T09:33:57.595 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-09T09:33:57.721 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-09T09:33:57.732 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-09T09:33:57.868 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-09T09:33:57.869 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-09T09:33:58.005 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-09T09:33:58.005 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-09T09:33:58.151 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-09T09:33:58.153 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-09T09:33:58.277 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-09T09:33:58.288 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-09T09:33:58.424 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-09T09:33:58.425 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-09T09:33:58.560 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-09T09:33:58.571 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-09T09:33:58.707 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-09T09:33:58.718 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-09T09:33:58.845 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-09T09:33:58.856 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-09T09:33:58.992 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-09T09:33:58.994 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-09T09:33:59.130 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-09T09:33:59.131 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-09T09:33:59.260 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-09T09:33:59.262 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-09T09:33:59.406 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-09T09:33:59.407 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-09T09:33:59.543 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-09T09:33:59.545 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-09T09:33:59.679 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-09T09:33:59.690 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-09T09:33:59.815 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-09T09:33:59.827 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-09T09:33:59.951 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-09T09:33:59.952 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-09T09:34:00.088 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-09T09:34:00.100 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-09T09:34:00.215 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-09T09:34:00.237 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-09T09:34:00.351 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-09T09:34:00.384 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-09T09:34:00.488 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-09T09:34:00.520 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-09T09:34:00.625 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-09T09:34:00.657 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-09T09:34:00.771 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-09T09:34:00.804 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-09T09:34:00.910 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-09T09:34:00.942 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-09T09:34:01.037 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-09T09:34:01.089 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-09T09:34:01.173 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-09T09:34:01.227 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-09T09:34:01.310 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-09T09:34:01.364 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-09T09:34:01.449 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-09T09:34:01.504 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-09T09:34:01.588 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-09T09:34:01.641 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-09T09:34:01.725 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-09T09:34:01.789 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-09T09:34:01.873 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-09T09:34:01.926 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-09T09:34:02.010 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-09T09:34:02.058 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:02 +0000 (0:00:06.820) 0:00:56.742 ********* 2021-03-09T09:34:02.077 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-09T09:34:02.202 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-09T09:34:02.308 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:02 +0000 (0:00:00.197) 0:00:56.939 ********* 2021-03-09T09:34:02.513 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-09T09:34:02.515 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282442.1481564-7980-115535165428910/source", "state": "file", "uid": 0} 2021-03-09T09:34:02.540 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:02 +0000 (0:00:00.285) 0:00:57.225 ********* 2021-03-09T09:34:02.621 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-09T09:34:02.623 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => { "msg": "Host smithi194.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-09T09:34:02.646 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:02 +0000 (0:00:00.105) 0:00:57.330 ********* 2021-03-09T09:34:02.708 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-09T09:34:02.710 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282442.3622515-7985-130384460577973/source", "state": "file", "uid": 0} 2021-03-09T09:34:02.732 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:02 +0000 (0:00:00.085) 0:00:57.416 ********* 2021-03-09T09:34:02.739 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-09T09:34:02.741 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-09T09:34:02.776 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:02 +0000 (0:00:00.043) 0:00:57.460 ********* 2021-03-09T09:34:02.827 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-09T09:34:02.829 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => { "msg": "Host smithi052.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-09T09:34:02.852 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:02 +0000 (0:00:00.076) 0:00:57.536 ********* 2021-03-09T09:34:02.858 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-03-09T09:34:02.860 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:02.884 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:02 +0000 (0:00:00.031) 0:00:57.568 ********* 2021-03-09T09:34:02.921 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-09T09:34:02.923 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-09T09:34:02.947 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:02 +0000 (0:00:00.063) 0:00:57.631 ********* 2021-03-09T09:34:02.965 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-09T09:34:02.967 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:02.993 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:02 +0000 (0:00:00.046) 0:00:57.677 ********* 2021-03-09T09:34:03.021 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-03-09T09:34:03.023 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.046 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.052) 0:00:57.730 ********* 2021-03-09T09:34:03.063 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-09T09:34:03.065 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.090 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.043) 0:00:57.774 ********* 2021-03-09T09:34:03.108 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-09T09:34:03.110 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.134 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.043) 0:00:57.818 ********* 2021-03-09T09:34:03.162 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-09T09:34:03.163 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.188 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.053) 0:00:57.872 ********* 2021-03-09T09:34:03.204 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-09T09:34:03.206 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.230 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.041) 0:00:57.914 ********* 2021-03-09T09:34:03.268 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-09T09:34:03.270 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.294 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.064) 0:00:57.978 ********* 2021-03-09T09:34:03.322 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-09T09:34:03.324 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.353 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.058) 0:00:58.037 ********* 2021-03-09T09:34:03.371 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-09T09:34:03.373 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.399 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.046) 0:00:58.083 ********* 2021-03-09T09:34:03.430 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-09T09:34:03.432 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.456 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.056) 0:00:58.140 ********* 2021-03-09T09:34:03.473 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-09T09:34:03.475 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.498 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.041) 0:00:58.182 ********* 2021-03-09T09:34:03.527 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-09T09:34:03.529 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.563 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.065) 0:00:58.247 ********* 2021-03-09T09:34:03.595 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-09T09:34:03.596 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.623 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.059) 0:00:58.307 ********* 2021-03-09T09:34:03.643 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-09T09:34:03.644 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.680 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.057) 0:00:58.364 ********* 2021-03-09T09:34:03.712 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-09T09:34:03.714 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:03.723 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:03.748 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.067) 0:00:58.432 ********* 2021-03-09T09:34:03.753 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-09T09:34:03.755 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.787 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.039) 0:00:58.471 ********* 2021-03-09T09:34:03.819 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.848 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.061) 0:00:58.533 ********* 2021-03-09T09:34:03.865 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-09T09:34:03.867 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:03.892 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.043) 0:00:58.576 ********* 2021-03-09T09:34:03.920 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-09T09:34:03.922 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.945 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.053) 0:00:58.629 ********* 2021-03-09T09:34:03.962 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-09T09:34:03.964 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:03.988 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:03 +0000 (0:00:00.043) 0:00:58.672 ********* 2021-03-09T09:34:04.015 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-09T09:34:04.017 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:04.046 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.057) 0:00:58.730 ********* 2021-03-09T09:34:04.067 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-09T09:34:04.069 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.095 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.048) 0:00:58.779 ********* 2021-03-09T09:34:04.124 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-09T09:34:04.126 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.148 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.053) 0:00:58.832 ********* 2021-03-09T09:34:04.166 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-09T09:34:04.167 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:04.192 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.044) 0:00:58.877 ********* 2021-03-09T09:34:04.220 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-09T09:34:04.222 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:04.249 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.056) 0:00:58.933 ********* 2021-03-09T09:34:04.268 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-09T09:34:04.270 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.303 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.054) 0:00:58.987 ********* 2021-03-09T09:34:04.346 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-09T09:34:04.348 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.371 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.067) 0:00:59.055 ********* 2021-03-09T09:34:04.378 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-09T09:34:04.379 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:04.408 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.037) 0:00:59.092 ********* 2021-03-09T09:34:04.448 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-09T09:34:04.450 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.484 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.075) 0:00:59.168 ********* 2021-03-09T09:34:04.492 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-09T09:34:04.493 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.517 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.032) 0:00:59.201 ********* 2021-03-09T09:34:04.547 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-09T09:34:04.549 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.587 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.069) 0:00:59.271 ********* 2021-03-09T09:34:04.603 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-09T09:34:04.605 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.628 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.041) 0:00:59.312 ********* 2021-03-09T09:34:04.656 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-09T09:34:04.658 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.680 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.051) 0:00:59.364 ********* 2021-03-09T09:34:04.698 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-09T09:34:04.700 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.723 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.042) 0:00:59.407 ********* 2021-03-09T09:34:04.761 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-09T09:34:04.762 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:04.785 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.062) 0:00:59.469 ********* 2021-03-09T09:34:04.802 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-09T09:34:04.804 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.827 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.042) 0:00:59.511 ********* 2021-03-09T09:34:04.855 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T09:34:04.857 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.883 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.055) 0:00:59.567 ********* 2021-03-09T09:34:04.900 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-09T09:34:04.901 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:04.925 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.042) 0:00:59.609 ********* 2021-03-09T09:34:04.953 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-09T09:34:04.955 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:04.977 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:04 +0000 (0:00:00.052) 0:00:59.662 ********* 2021-03-09T09:34:04.994 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T09:34:04.996 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.019 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.041) 0:00:59.703 ********* 2021-03-09T09:34:05.047 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-09T09:34:05.049 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.072 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.052) 0:00:59.756 ********* 2021-03-09T09:34:05.088 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-09T09:34:05.090 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.113 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.041) 0:00:59.797 ********* 2021-03-09T09:34:05.141 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.165 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.052) 0:00:59.849 ********* 2021-03-09T09:34:05.183 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-09T09:34:05.184 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.207 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.041) 0:00:59.891 ********* 2021-03-09T09:34:05.234 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-09T09:34:05.236 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.263 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.055) 0:00:59.947 ********* 2021-03-09T09:34:05.279 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-09T09:34:05.281 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.304 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.041) 0:00:59.988 ********* 2021-03-09T09:34:05.333 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-09T09:34:05.334 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.359 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.054) 0:01:00.043 ********* 2021-03-09T09:34:05.377 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-09T09:34:05.378 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.402 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.042) 0:01:00.086 ********* 2021-03-09T09:34:05.441 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-09T09:34:05.442 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:05.469 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.066) 0:01:00.153 ********* 2021-03-09T09:34:05.497 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-09T09:34:05.498 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.521 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.052) 0:01:00.206 ********* 2021-03-09T09:34:05.540 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-09T09:34:05.542 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.564 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.042) 0:01:00.248 ********* 2021-03-09T09:34:05.612 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-09T09:34:05.614 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:05.638 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.073) 0:01:00.322 ********* 2021-03-09T09:34:05.643 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-09T09:34:05.645 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.672 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.033) 0:01:00.356 ********* 2021-03-09T09:34:05.699 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-09T09:34:05.701 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.725 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.053) 0:01:00.409 ********* 2021-03-09T09:34:05.744 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-09T09:34:05.746 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.781 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.056) 0:01:00.465 ********* 2021-03-09T09:34:05.812 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-09T09:34:05.814 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.837 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.056) 0:01:00.522 ********* 2021-03-09T09:34:05.875 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-09T09:34:05.877 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.901 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.063) 0:01:00.585 ********* 2021-03-09T09:34:05.918 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-09T09:34:05.919 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:05.951 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:05 +0000 (0:00:00.049) 0:01:00.635 ********* 2021-03-09T09:34:05.981 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-09T09:34:05.983 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.007 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.056) 0:01:00.691 ********* 2021-03-09T09:34:06.057 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-09T09:34:06.059 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.093 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.086) 0:01:00.777 ********* 2021-03-09T09:34:06.100 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T09:34:06.101 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.129 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.035) 0:01:00.813 ********* 2021-03-09T09:34:06.169 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-09T09:34:06.170 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.172 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.208 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.079) 0:01:00.892 ********* 2021-03-09T09:34:06.280 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T09:34:06.281 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.303 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.094) 0:01:00.987 ********* 2021-03-09T09:34:06.362 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T09:34:06.364 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.021404", "end": "2021-03-09 09:34:06.302695", "rc": 0, "start": "2021-03-09 09:34:06.281291", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-09T09:34:06.387 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.083) 0:01:01.071 ********* 2021-03-09T09:34:06.461 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-09T09:34:06.463 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.491 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.104) 0:01:01.175 ********* 2021-03-09T09:34:06.508 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T09:34:06.510 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.013387", "end": "2021-03-09 09:34:06.479843", "rc": 0, "start": "2021-03-09 09:34:06.466456", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-09T09:34:06.535 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.043) 0:01:01.219 ********* 2021-03-09T09:34:06.562 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-09T09:34:06.564 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.587 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.052) 0:01:01.271 ********* 2021-03-09T09:34:06.604 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-09T09:34:06.606 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.629 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.042) 0:01:01.314 ********* 2021-03-09T09:34:06.668 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T09:34:06.669 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.680 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.682 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.691 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.715 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.085) 0:01:01.399 ********* 2021-03-09T09:34:06.721 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-09T09:34:06.723 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.746 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.031) 0:01:01.431 ********* 2021-03-09T09:34:06.784 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T09:34:06.786 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.808 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.061) 0:01:01.492 ********* 2021-03-09T09:34:06.825 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T09:34:06.827 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.836 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.838 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.848 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.874 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.065) 0:01:01.558 ********* 2021-03-09T09:34:06.880 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-09T09:34:06.882 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.909 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.034) 0:01:01.593 ********* 2021-03-09T09:34:06.947 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T09:34:06.948 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:06.971 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.062) 0:01:01.656 ********* 2021-03-09T09:34:06.978 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-09T09:34:06.979 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:07.004 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:06 +0000 (0:00:00.031) 0:01:01.688 ********* 2021-03-09T09:34:07.041 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-09T09:34:07.043 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:07.066 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.062) 0:01:01.750 ********* 2021-03-09T09:34:07.083 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-09T09:34:07.085 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:07.109 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.042) 0:01:01.793 ********* 2021-03-09T09:34:07.137 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-09T09:34:07.139 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:07.162 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.053) 0:01:01.846 ********* 2021-03-09T09:34:07.179 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-09T09:34:07.180 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:07.203 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.041) 0:01:01.887 ********* 2021-03-09T09:34:07.230 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-09T09:34:07.232 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:07.256 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.052) 0:01:01.940 ********* 2021-03-09T09:34:07.336 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-09T09:34:07.338 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:07.360 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.104) 0:01:02.044 ********* 2021-03-09T09:34:07.522 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-09T09:34:07.524 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.019376", "end": "2021-03-09 09:34:07.500350", "rc": 0, "start": "2021-03-09 09:34:07.480974", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:34:07.549 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.189) 0:01:02.233 ********* 2021-03-09T09:34:07.567 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.215817", "end": "2021-03-09 09:34:07.500400", "rc": 0, "start": "2021-03-09 09:34:07.284583", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:34:07.592 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.042) 0:01:02.276 ********* 2021-03-09T09:34:07.631 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-09T09:34:07.633 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:07.657 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.064) 0:01:02.341 ********* 2021-03-09T09:34:07.695 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:07.723 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.066) 0:01:02.408 ********* 2021-03-09T09:34:07.867 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-09T09:34:07.869 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-03-09T09:34:07.869 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.015780", "end": "2021-03-09 09:34:07.850444", "rc": 0, "start": "2021-03-09 09:34:07.834664", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:34:07.894 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.170) 0:01:02.578 ********* 2021-03-09T09:34:07.922 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.024760", "end": "2021-03-09 09:34:07.868766", "rc": 0, "start": "2021-03-09 09:34:07.844006", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:34:07.947 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:07 +0000 (0:00:00.052) 0:01:02.631 ********* 2021-03-09T09:34:11.222 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-09T09:34:11.223 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:34:11.246 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:11 +0000 (0:00:03.299) 0:01:05.930 ********* 2021-03-09T09:34:11.253 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:34:11.279 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:11 +0000 (0:00:00.032) 0:01:05.963 ********* 2021-03-09T09:34:12.144 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-09T09:34:12.146 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:34:12.168 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:12 +0000 (0:00:00.888) 0:01:06.852 ********* 2021-03-09T09:34:12.216 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:34:12.240 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:12 +0000 (0:00:00.071) 0:01:06.924 ********* 2021-03-09T09:34:12.246 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-09T09:34:12.248 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:12.272 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:12 +0000 (0:00:00.031) 0:01:06.956 ********* 2021-03-09T09:34:12.310 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:12.335 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:12 +0000 (0:00:00.062) 0:01:07.019 ********* 2021-03-09T09:34:12.956 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-09T09:34:12.958 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-09T09:34:12.980 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:12 +0000 (0:00:00.645) 0:01:07.664 ********* 2021-03-09T09:34:13.001 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:12 +0000 (0:00:00.020) 0:01:07.685 ********* 2021-03-09T09:34:13.916 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-09T09:34:13.918 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:34:13.939 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:13 +0000 (0:00:00.938) 0:01:08.623 ********* 2021-03-09T09:34:13.961 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:13 +0000 (0:00:00.021) 0:01:08.645 ********* 2021-03-09T09:34:14.342 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T09:34:14.344 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-09T09:34:14.520 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-09T09:34:14.647 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-09T09:34:14.826 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-09T09:34:14.932 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-09T09:34:14.961 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:14 +0000 (0:00:00.999) 0:01:09.645 ********* 2021-03-09T09:34:15.092 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-09T09:34:15.093 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:34:15.117 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:15 +0000 (0:00:00.156) 0:01:09.801 ********* 2021-03-09T09:34:15.123 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T09:34:15.125 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-09T09:34:15.153 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:15 +0000 (0:00:00.035) 0:01:09.837 ********* 2021-03-09T09:34:15.296 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-09T09:34:15.297 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:34:15.320 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:15 +0000 (0:00:00.167) 0:01:10.004 ********* 2021-03-09T09:34:15.537 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T09:34:15.539 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-09T09:34:15.694 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-09T09:34:15.821 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-09T09:34:15.972 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-09T09:34:16.109 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-09T09:34:16.136 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.815) 0:01:10.820 ********* 2021-03-09T09:34:16.216 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-09T09:34:16.217 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-09T09:34:16.239 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.103) 0:01:10.923 ********* 2021-03-09T09:34:16.256 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T09:34:16.257 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-09T09:34:16.286 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.043) 0:01:10.966 ********* 2021-03-09T09:34:16.309 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.342 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.059) 0:01:11.026 ********* 2021-03-09T09:34:16.360 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-09T09:34:16.362 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-09T09:34:16.386 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.043) 0:01:11.070 ********* 2021-03-09T09:34:16.412 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-09T09:34:16.414 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:16.438 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.052) 0:01:11.122 ********* 2021-03-09T09:34:16.466 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-09T09:34:16.468 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.491 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.052) 0:01:11.175 ********* 2021-03-09T09:34:16.539 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-09T09:34:16.541 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.550 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.552 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.562 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.585 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.094) 0:01:11.269 ********* 2021-03-09T09:34:16.592 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-09T09:34:16.594 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:16.617 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.031) 0:01:11.301 ********* 2021-03-09T09:34:16.654 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-09T09:34:16.656 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.678 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.061) 0:01:11.363 ********* 2021-03-09T09:34:16.695 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-09T09:34:16.697 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.706 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.708 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.718 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.742 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.063) 0:01:11.426 ********* 2021-03-09T09:34:16.811 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-09T09:34:16.812 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:16.838 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:16 +0000 (0:00:00.095) 0:01:11.522 ********* 2021-03-09T09:34:17.636 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-09T09:34:17.638 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:34:17.660 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:17 +0000 (0:00:00.822) 0:01:12.344 ********* 2021-03-09T09:34:17.782 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:34:17.807 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:17 +0000 (0:00:00.146) 0:01:12.491 ********* 2021-03-09T09:34:18.042 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-09T09:34:18.044 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-09T09:34:18.067 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:18 +0000 (0:00:00.259) 0:01:12.751 ********* 2021-03-09T09:34:18.167 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-09T09:34:18.191 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:18 +0000 (0:00:00.124) 0:01:12.876 ********* 2021-03-09T09:34:18.438 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-09T09:34:18.440 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-09T09:34:18.464 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:18 +0000 (0:00:00.272) 0:01:13.148 ********* 2021-03-09T09:34:18.566 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-09T09:34:18.590 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:18 +0000 (0:00:00.126) 0:01:13.274 ********* 2021-03-09T09:34:18.659 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T09:34:18.661 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-09T09:34:18.775 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-09T09:34:18.777 INFO:teuthology.task.ansible.out: 2021-03-09T09:34:18.803 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:18 +0000 (0:00:00.213) 0:01:13.488 ********* 2021-03-09T09:34:18.811 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-09T09:34:18.875 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-09T09:34:18.876 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:18.902 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:18 +0000 (0:00:00.098) 0:01:13.587 ********* 2021-03-09T09:34:18.910 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T09:34:18.912 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-09T09:34:18.938 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:18 +0000 (0:00:00.035) 0:01:13.622 ********* 2021-03-09T09:34:18.976 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-09T09:34:18.978 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:19.001 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:18 +0000 (0:00:00.062) 0:01:13.685 ********* 2021-03-09T09:34:19.019 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:19.044 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:19 +0000 (0:00:00.042) 0:01:13.728 ********* 2021-03-09T09:34:19.072 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-09T09:34:19.074 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:19.097 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:19 +0000 (0:00:00.053) 0:01:13.781 ********* 2021-03-09T09:34:19.115 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-09T09:34:19.116 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:34:19.140 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:19 +0000 (0:00:00.043) 0:01:13.824 ********* 2021-03-09T09:34:19.354 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:19.381 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:19 +0000 (0:00:00.240) 0:01:14.065 ********* 2021-03-09T09:34:19.647 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-09T09:34:19.649 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-09T09:34:19.678 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:19 +0000 (0:00:00.297) 0:01:14.362 ********* 2021-03-09T09:34:19.886 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-09T09:34:19.911 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:19 +0000 (0:00:00.232) 0:01:14.595 ********* 2021-03-09T09:34:20.341 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-09T09:34:20.342 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:31:49 UTC", "ActiveEnterTimestampMonotonic": "17472460", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket network.target -.mount nss-lookup.target local-fs.target var-run.mount system.slice time-sync.target sysinit.target remote-fs.target systemd-tmpfiles-setup.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:31:49 UTC", "AssertTimestampMonotonic": "17471139", "Before": "shutdown.target multi-user.target xdm.service plymouth-quit.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:31:49 UTC", "ConditionTimestampMonotonic": "17471139", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-09T09:34:20.344 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1374", "ExecMainStartTimestamp": "Tue 2021-03-09 09:31:49 UTC", "ExecMainStartTimestampMonotonic": "17472420", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:31:49 UTC", "InactiveExitTimestampMonotonic": "17472460", "InvocationID": "832c2dba90464e4e84e33177bc363825", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2021-03-09T09:34:20.345 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1374", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2021-03-09T09:34:20.347 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:31:49 UTC", "StateChangeTimestampMonotonic": "17472460", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:31:49 UTC", "WatchdogTimestampMonotonic": "17472459", "WatchdogUSec": "0"}} 2021-03-09T09:34:20.466 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.554) 0:01:15.150 ********* 2021-03-09T09:34:20.476 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:32:16 UTC", "ActiveEnterTimestampMonotonic": "17325560", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target systemd-journald.socket network.target systemd-tmpfiles-setup.service sysinit.target local-fs.target remote-fs.target system.slice nss-lookup.target var-run.mount basic.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:32:16 UTC", "AssertTimestampMonotonic": "17324560", "Before": "shutdown.target xdm.service plymouth-quit.service getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:32:16 UTC", "ConditionTimestampMonotonic": "17324560", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-09T09:34:20.478 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1164", "ExecMainStartTimestamp": "Tue 2021-03-09 09:32:16 UTC", "ExecMainStartTimestampMonotonic": "17325527", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:32:16 UTC", "InactiveExitTimestampMonotonic": "17325560", "InvocationID": "ad79d0db87eb45cdaac075ca1732a439", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2021-03-09T09:34:20.479 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1164", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2021-03-09T09:34:20.481 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:32:16 UTC", "StateChangeTimestampMonotonic": "17325560", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:32:16 UTC", "WatchdogTimestampMonotonic": "17325559", "WatchdogUSec": "0"}} 2021-03-09T09:34:20.504 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.037) 0:01:15.188 ********* 2021-03-09T09:34:20.542 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-09T09:34:20.543 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:20.567 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.063) 0:01:15.251 ********* 2021-03-09T09:34:20.585 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:20.610 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.042) 0:01:15.294 ********* 2021-03-09T09:34:20.637 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-09T09:34:20.639 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:20.664 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.053) 0:01:15.348 ********* 2021-03-09T09:34:20.681 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:20.707 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.043) 0:01:15.391 ********* 2021-03-09T09:34:20.735 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-09T09:34:20.737 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:20.762 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.054) 0:01:15.446 ********* 2021-03-09T09:34:20.791 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:20.816 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.053) 0:01:15.500 ********* 2021-03-09T09:34:20.843 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-09T09:34:20.845 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:20.869 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.052) 0:01:15.553 ********* 2021-03-09T09:34:20.896 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:20.920 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.050) 0:01:15.604 ********* 2021-03-09T09:34:20.936 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-09T09:34:20.938 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:20.961 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:20 +0000 (0:00:00.041) 0:01:15.645 ********* 2021-03-09T09:34:20.988 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.016 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.054) 0:01:15.700 ********* 2021-03-09T09:34:21.032 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-09T09:34:21.034 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.057 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.041) 0:01:15.742 ********* 2021-03-09T09:34:21.085 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.109 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.051) 0:01:15.793 ********* 2021-03-09T09:34:21.125 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-09T09:34:21.127 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.151 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.041) 0:01:15.835 ********* 2021-03-09T09:34:21.189 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.216 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.064) 0:01:15.900 ********* 2021-03-09T09:34:21.234 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-09T09:34:21.235 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.260 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.044) 0:01:15.945 ********* 2021-03-09T09:34:21.300 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.325 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.064) 0:01:16.009 ********* 2021-03-09T09:34:21.343 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-09T09:34:21.345 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.369 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.043) 0:01:16.053 ********* 2021-03-09T09:34:21.411 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.444 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.074) 0:01:16.128 ********* 2021-03-09T09:34:21.463 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-09T09:34:21.465 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.488 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.044) 0:01:16.172 ********* 2021-03-09T09:34:21.527 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:21.551 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.062) 0:01:16.235 ********* 2021-03-09T09:34:21.579 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-09T09:34:21.581 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-09T09:34:21.604 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.052) 0:01:16.288 ********* 2021-03-09T09:34:21.632 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-09T09:34:21.660 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.056) 0:01:16.344 ********* 2021-03-09T09:34:21.698 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-09T09:34:21.700 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-09T09:34:21.724 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.064) 0:01:16.408 ********* 2021-03-09T09:34:21.742 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-09T09:34:21.768 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.043) 0:01:16.452 ********* 2021-03-09T09:34:21.806 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-09T09:34:21.808 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-09T09:34:21.832 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.063) 0:01:16.516 ********* 2021-03-09T09:34:21.849 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-09T09:34:21.875 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.043) 0:01:16.560 ********* 2021-03-09T09:34:21.914 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-09T09:34:21.916 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T09:34:21.940 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.064) 0:01:16.624 ********* 2021-03-09T09:34:21.967 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T09:34:21.993 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:21 +0000 (0:00:00.053) 0:01:16.677 ********* 2021-03-09T09:34:22.082 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-09T09:34:22.084 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T09:34:22.107 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:22 +0000 (0:00:00.114) 0:01:16.791 ********* 2021-03-09T09:34:22.145 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T09:34:22.170 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:22 +0000 (0:00:00.062) 0:01:16.854 ********* 2021-03-09T09:34:22.292 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-09T09:34:22.316 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:22 +0000 (0:00:00.145) 0:01:17.000 ********* 2021-03-09T09:34:22.364 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-09T09:34:22.388 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:22 +0000 (0:00:00.072) 0:01:17.072 ********* 2021-03-09T09:34:22.394 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-09T09:34:22.396 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:22.420 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:22 +0000 (0:00:00.031) 0:01:17.104 ********* 2021-03-09T09:34:22.457 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:22.486 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:22 +0000 (0:00:00.065) 0:01:17.170 ********* 2021-03-09T09:34:22.503 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-09T09:34:22.504 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:22.528 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:22 +0000 (0:00:00.042) 0:01:17.212 ********* 2021-03-09T09:34:22.556 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:22.580 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:22 +0000 (0:00:00.051) 0:01:17.264 ********* 2021-03-09T09:34:22.742 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-09T09:34:22.744 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-09T09:34:22.768 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:22 +0000 (0:00:00.187) 0:01:17.452 ********* 2021-03-09T09:34:22.796 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-09T09:34:22.821 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:22 +0000 (0:00:00.053) 0:01:17.505 ********* 2021-03-09T09:34:23.234 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-09T09:34:23.236 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-09T09:34:23.263 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:23 +0000 (0:00:00.441) 0:01:17.947 ********* 2021-03-09T09:34:23.281 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-09T09:34:23.308 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:23 +0000 (0:00:00.045) 0:01:17.992 ********* 2021-03-09T09:34:23.346 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-09T09:34:23.348 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi052"}, "changed": false} 2021-03-09T09:34:23.373 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:23 +0000 (0:00:00.064) 0:01:18.057 ********* 2021-03-09T09:34:23.390 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi194"}, "changed": false} 2021-03-09T09:34:23.420 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:23 +0000 (0:00:00.046) 0:01:18.104 ********* 2021-03-09T09:34:24.334 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-09T09:34:24.336 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi194", "ansible_hostname": "smithi194", "ansible_nodename": "smithi194"}, "changed": false, "name": "smithi194"} 2021-03-09T09:34:24.361 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:24 +0000 (0:00:00.941) 0:01:19.045 ********* 2021-03-09T09:34:24.368 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi052", "ansible_hostname": "smithi052", "ansible_nodename": "smithi052"}, "changed": false, "name": "smithi052"} 2021-03-09T09:34:24.397 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:24 +0000 (0:00:00.035) 0:01:19.081 ********* 2021-03-09T09:34:24.761 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-09T09:34:24.763 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-09T09:34:24.786 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:24 +0000 (0:00:00.388) 0:01:19.470 ********* 2021-03-09T09:34:24.793 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-09T09:34:24.820 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:24 +0000 (0:00:00.034) 0:01:19.504 ********* 2021-03-09T09:34:25.159 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-09T09:34:25.161 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-09T09:34:25.183 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:25 +0000 (0:00:00.363) 0:01:19.867 ********* 2021-03-09T09:34:25.211 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-09T09:34:25.235 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:25 +0000 (0:00:00.052) 0:01:19.919 ********* 2021-03-09T09:34:25.661 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-09T09:34:25.663 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-09T09:34:25.687 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:25 +0000 (0:00:00.451) 0:01:20.371 ********* 2021-03-09T09:34:25.705 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-09T09:34:25.735 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:25 +0000 (0:00:00.048) 0:01:20.419 ********* 2021-03-09T09:34:25.755 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-09T09:34:25.757 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:25.783 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:25 +0000 (0:00:00.047) 0:01:20.467 ********* 2021-03-09T09:34:25.810 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:25.844 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:25 +0000 (0:00:00.061) 0:01:20.528 ********* 2021-03-09T09:34:25.861 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T09:34:25.863 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:25.887 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:25 +0000 (0:00:00.042) 0:01:20.571 ********* 2021-03-09T09:34:25.914 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:25.938 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:25 +0000 (0:00:00.051) 0:01:20.622 ********* 2021-03-09T09:34:25.955 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T09:34:25.957 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:25.981 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:25 +0000 (0:00:00.042) 0:01:20.665 ********* 2021-03-09T09:34:26.010 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.036 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.055) 0:01:20.720 ********* 2021-03-09T09:34:26.053 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T09:34:26.055 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.079 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.043) 0:01:20.763 ********* 2021-03-09T09:34:26.117 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.142 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.062) 0:01:20.826 ********* 2021-03-09T09:34:26.160 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T09:34:26.162 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.187 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.045) 0:01:20.871 ********* 2021-03-09T09:34:26.217 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.243 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.056) 0:01:20.927 ********* 2021-03-09T09:34:26.260 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T09:34:26.262 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.285 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.041) 0:01:20.969 ********* 2021-03-09T09:34:26.324 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.363 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.078) 0:01:21.047 ********* 2021-03-09T09:34:26.372 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.405 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.040) 0:01:21.088 ********* 2021-03-09T09:34:26.444 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.471 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.067) 0:01:21.156 ********* 2021-03-09T09:34:26.489 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-09T09:34:26.491 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.516 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.044) 0:01:21.200 ********* 2021-03-09T09:34:26.545 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.571 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.054) 0:01:21.255 ********* 2021-03-09T09:34:26.589 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-09T09:34:26.626 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.054) 0:01:21.310 ********* 2021-03-09T09:34:26.683 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.057) 0:01:21.367 ********* 2021-03-09T09:34:26.705 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-09T09:34:26.731 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.047) 0:01:21.415 ********* 2021-03-09T09:34:26.787 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.055) 0:01:21.471 ********* 2021-03-09T09:34:26.867 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.047) 0:01:21.518 ********* 2021-03-09T09:34:26.899 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.064) 0:01:21.583 ********* 2021-03-09T09:34:26.917 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-09T09:34:26.919 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:26.946 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.046) 0:01:21.630 ********* 2021-03-09T09:34:26.975 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.000 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:26 +0000 (0:00:00.054) 0:01:21.684 ********* 2021-03-09T09:34:27.030 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-09T09:34:27.032 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.042 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.072 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.070) 0:01:21.755 ********* 2021-03-09T09:34:27.090 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.091 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.120 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.049) 0:01:21.804 ********* 2021-03-09T09:34:27.140 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-09T09:34:27.142 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.166 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.045) 0:01:21.850 ********* 2021-03-09T09:34:27.193 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.219 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.052) 0:01:21.903 ********* 2021-03-09T09:34:27.257 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:34:27.259 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.283 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.064) 0:01:21.967 ********* 2021-03-09T09:34:27.289 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.323 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.039) 0:01:22.007 ********* 2021-03-09T09:34:27.362 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-09T09:34:27.364 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.392 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.068) 0:01:22.076 ********* 2021-03-09T09:34:27.398 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.432 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.035) 0:01:22.111 ********* 2021-03-09T09:34:27.464 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-09T09:34:27.466 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.490 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.063) 0:01:22.174 ********* 2021-03-09T09:34:27.517 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.542 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.052) 0:01:22.226 ********* 2021-03-09T09:34:27.570 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-09T09:34:27.572 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.595 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.052) 0:01:22.279 ********* 2021-03-09T09:34:27.623 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.647 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.052) 0:01:22.331 ********* 2021-03-09T09:34:27.676 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-09T09:34:27.678 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.705 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.057) 0:01:22.389 ********* 2021-03-09T09:34:27.755 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.781 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.075) 0:01:22.465 ********* 2021-03-09T09:34:27.788 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T09:34:27.789 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.815 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.033) 0:01:22.499 ********* 2021-03-09T09:34:27.843 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.868 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.052) 0:01:22.552 ********* 2021-03-09T09:34:27.885 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T09:34:27.886 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.910 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.042) 0:01:22.594 ********* 2021-03-09T09:34:27.940 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:27.965 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:27 +0000 (0:00:00.054) 0:01:22.649 ********* 2021-03-09T09:34:27.983 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T09:34:27.985 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.008 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.043) 0:01:22.692 ********* 2021-03-09T09:34:28.036 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.061 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.052) 0:01:22.745 ********* 2021-03-09T09:34:28.079 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.104 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.043) 0:01:22.788 ********* 2021-03-09T09:34:28.133 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.158 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.053) 0:01:22.842 ********* 2021-03-09T09:34:28.175 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T09:34:28.177 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.201 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.043) 0:01:22.885 ********* 2021-03-09T09:34:28.250 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.274 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.073) 0:01:22.958 ********* 2021-03-09T09:34:28.302 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T09:34:28.304 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.331 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.056) 0:01:23.015 ********* 2021-03-09T09:34:28.348 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.375 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.043) 0:01:23.059 ********* 2021-03-09T09:34:28.413 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-09T09:34:28.415 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.439 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.064) 0:01:23.123 ********* 2021-03-09T09:34:28.458 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.495 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.055) 0:01:23.179 ********* 2021-03-09T09:34:28.526 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-09T09:34:28.528 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.552 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.057) 0:01:23.237 ********* 2021-03-09T09:34:28.571 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.596 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.043) 0:01:23.280 ********* 2021-03-09T09:34:28.627 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-09T09:34:28.629 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.662 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.065) 0:01:23.346 ********* 2021-03-09T09:34:28.683 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.721 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.058) 0:01:23.405 ********* 2021-03-09T09:34:28.739 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-09T09:34:28.741 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.764 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.043) 0:01:23.448 ********* 2021-03-09T09:34:28.802 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.826 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.061) 0:01:23.510 ********* 2021-03-09T09:34:28.844 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-09T09:34:28.846 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.870 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.043) 0:01:23.554 ********* 2021-03-09T09:34:28.898 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.925 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.054) 0:01:23.609 ********* 2021-03-09T09:34:28.944 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-09T09:34:28.946 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:28.976 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:28 +0000 (0:00:00.051) 0:01:23.660 ********* 2021-03-09T09:34:29.005 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:29.031 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.055) 0:01:23.715 ********* 2021-03-09T09:34:29.048 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-09T09:34:29.050 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:29.074 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.042) 0:01:23.758 ********* 2021-03-09T09:34:29.101 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:29.125 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.051) 0:01:23.810 ********* 2021-03-09T09:34:29.143 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-09T09:34:29.144 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:29.168 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.042) 0:01:23.852 ********* 2021-03-09T09:34:29.196 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:29.221 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.052) 0:01:23.905 ********* 2021-03-09T09:34:29.365 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-09T09:34:29.367 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005209", "end": "2021-03-09 09:34:29.314316", "rc": 0, "start": "2021-03-09 09:34:29.309107", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:34:29.393 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.172) 0:01:24.077 ********* 2021-03-09T09:34:29.411 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004528", "end": "2021-03-09 09:34:29.397097", "rc": 0, "start": "2021-03-09 09:34:29.392569", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:34:29.436 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.043) 0:01:24.120 ********* 2021-03-09T09:34:29.475 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-09T09:34:29.500 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.063) 0:01:24.184 ********* 2021-03-09T09:34:29.545 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.045) 0:01:24.229 ********* 2021-03-09T09:34:29.889 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-09T09:34:29.891 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-09T09:34:29.914 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.368) 0:01:24.598 ********* 2021-03-09T09:34:29.941 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-09T09:34:29.969 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:29 +0000 (0:00:00.055) 0:01:24.653 ********* 2021-03-09T09:34:29.987 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-09T09:34:29.989 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:30.012 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:30 +0000 (0:00:00.042) 0:01:24.696 ********* 2021-03-09T09:34:30.040 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:30.065 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:30 +0000 (0:00:00.052) 0:01:24.749 ********* 2021-03-09T09:34:31.426 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-09T09:34:31.428 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-09T09:34:31.459 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-09T09:34:32.359 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-09T09:34:32.389 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:32 +0000 (0:00:02.323) 0:01:27.073 ********* 2021-03-09T09:34:32.417 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-09T09:34:32.444 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:32 +0000 (0:00:00.054) 0:01:27.128 ********* 2021-03-09T09:34:33.341 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-09T09:34:33.343 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:34:33.365 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:33 +0000 (0:00:00.921) 0:01:28.049 ********* 2021-03-09T09:34:33.427 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:34:33.455 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:33 +0000 (0:00:00.089) 0:01:28.139 ********* 2021-03-09T09:34:33.464 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-09T09:34:33.494 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:33 +0000 (0:00:00.039) 0:01:28.178 ********* 2021-03-09T09:34:33.549 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:33 +0000 (0:00:00.054) 0:01:28.233 ********* 2021-03-09T09:34:33.713 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-09T09:34:33.714 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.037310", "end": "2021-03-09 09:34:33.651600", "rc": 0, "start": "2021-03-09 09:34:33.614290", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:34:33.737 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:33 +0000 (0:00:00.188) 0:01:28.421 ********* 2021-03-09T09:34:33.756 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.022120", "end": "2021-03-09 09:34:33.736966", "rc": 0, "start": "2021-03-09 09:34:33.714846", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:34:33.782 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:33 +0000 (0:00:00.044) 0:01:28.466 ********* 2021-03-09T09:34:36.721 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-09T09:34:36.723 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:34:36.744 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:36 +0000 (0:00:02.962) 0:01:31.429 ********* 2021-03-09T09:34:36.751 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:34:36.778 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:36 +0000 (0:00:00.033) 0:01:31.462 ********* 2021-03-09T09:34:47.018 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-09T09:34:47.020 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102064 files and directorie 2021-03-09T09:34:47.022 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-03-09T09:34:47.024 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102064 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T09:34:47.059 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:47 +0000 (0:00:10.280) 0:01:41.743 ********* 2021-03-09T09:34:47.320 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102064 files and directorie 2021-03-09T09:34:47.322 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-03-09T09:34:47.324 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102064 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T09:34:47.346 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:47 +0000 (0:00:00.286) 0:01:42.030 ********* 2021-03-09T09:34:48.001 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T09:34:48.003 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:34:48.027 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:48 +0000 (0:00:00.680) 0:01:42.711 ********* 2021-03-09T09:34:48.098 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-09T09:34:48.100 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:48.121 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:48 +0000 (0:00:00.094) 0:01:42.806 ********* 2021-03-09T09:34:48.318 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T09:34:48.320 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:34:48.343 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:48 +0000 (0:00:00.221) 0:01:43.027 ********* 2021-03-09T09:34:48.423 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-09T09:34:48.425 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:48.447 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:48 +0000 (0:00:00.103) 0:01:43.131 ********* 2021-03-09T09:34:56.830 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T09:34:56.832 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 10.4 MB in 1s (9868 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-03-09T09:34:56.833 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101922 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-03-09T09:34:56.835 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]", "Fetched 10.4 MB in 1s (9868 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-03-09T09:34:56.837 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101922 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T09:34:56.855 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:56 +0000 (0:00:08.408) 0:01:51.540 ********* 2021-03-09T09:34:57.355 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 10.4 MB in 1s (9017 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-03-09T09:34:57.357 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101922 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-03-09T09:34:57.358 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]", "Fetched 10.4 MB in 1s (9017 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-03-09T09:34:57.360 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101922 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T09:34:57.393 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:57 +0000 (0:00:00.537) 0:01:52.077 ********* 2021-03-09T09:34:57.771 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-09T09:34:57.773 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:34:57.795 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:57 +0000 (0:00:00.402) 0:01:52.480 ********* 2021-03-09T09:34:58.383 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:34:58.409 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:58 +0000 (0:00:00.612) 0:01:53.093 ********* 2021-03-09T09:34:58.716 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-09T09:34:58.718 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:34:58.741 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:58 +0000 (0:00:00.332) 0:01:53.425 ********* 2021-03-09T09:34:58.926 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-09T09:34:58.928 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-09T09:34:58.955 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:58 +0000 (0:00:00.214) 0:01:53.639 ********* 2021-03-09T09:34:59.027 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-09T09:34:59.029 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:59.051 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:59 +0000 (0:00:00.095) 0:01:53.735 ********* 2021-03-09T09:34:59.254 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-09T09:34:59.256 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-09T09:34:59.278 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:59 +0000 (0:00:00.227) 0:01:53.962 ********* 2021-03-09T09:34:59.363 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-09T09:34:59.365 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:34:59.389 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:59 +0000 (0:00:00.111) 0:01:54.073 ********* 2021-03-09T09:34:59.577 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-09T09:34:59.581 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-09T09:34:59.606 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:59 +0000 (0:00:00.216) 0:01:54.289 ********* 2021-03-09T09:34:59.699 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:34:59.724 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:59 +0000 (0:00:00.118) 0:01:54.408 ********* 2021-03-09T09:34:59.731 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-09T09:34:59.733 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-09T09:34:59.771 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:59 +0000 (0:00:00.047) 0:01:54.455 ********* 2021-03-09T09:34:59.928 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-09T09:34:59.930 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-09T09:34:59.953 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:34:59 +0000 (0:00:00.181) 0:01:54.637 ********* 2021-03-09T09:34:59.995 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi194.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-09T09:35:00.035 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:00 +0000 (0:00:00.082) 0:01:54.719 ********* 2021-03-09T09:35:00.231 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-09T09:35:00.233 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:35:00.257 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:00 +0000 (0:00:00.221) 0:01:54.941 ********* 2021-03-09T09:35:00.379 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-09T09:35:00.381 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-09T09:35:00.407 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:00 +0000 (0:00:00.150) 0:01:55.091 ********* 2021-03-09T09:35:00.593 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-09T09:35:00.595 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:32:16 UTC", "ActiveEnterTimestampMonotonic": "17672804", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target -.mount systemd-tmpfiles-setup.service remote-fs.target basic.target system.slice nss-lookup.target network.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:32:16 UTC", "AssertTimestampMonotonic": "17322859", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:32:16 UTC", "ConditionTimestampMonotonic": "17322858", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-09T09:35:00.596 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1228", "ExecMainStartTimestamp": "Tue 2021-03-09 09:32:16 UTC", "ExecMainStartTimestampMonotonic": "17672792", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:32:16 UTC", "InactiveExitTimestampMonotonic": "17323579", "InvocationID": "6c28da6057ad41ae903732c590d29fb7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-09T09:35:00.598 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1228", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-09T09:35:00.599 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:32:16 UTC", "StateChangeTimestampMonotonic": "17672804", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:32:16 UTC", "WatchdogTimestampMonotonic": "17672803", "WatchdogUSec": "0"}} 2021-03-09T09:35:00.619 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:00 +0000 (0:00:00.211) 0:01:55.303 ********* 2021-03-09T09:35:00.625 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-09T09:35:00.626 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-09T09:35:00.651 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:00 +0000 (0:00:00.032) 0:01:55.335 ********* 2021-03-09T09:35:00.689 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T09:35:00.691 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:00.714 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:00 +0000 (0:00:00.062) 0:01:55.398 ********* 2021-03-09T09:35:00.794 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T09:35:00.796 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:00.820 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:00 +0000 (0:00:00.105) 0:01:55.504 ********* 2021-03-09T09:35:00.837 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-09T09:35:00.839 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:35:00.872 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:00 +0000 (0:00:00.052) 0:01:55.556 ********* 2021-03-09T09:35:00.901 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T09:35:00.903 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:00.925 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:00 +0000 (0:00:00.053) 0:01:55.609 ********* 2021-03-09T09:35:01.004 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.029 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.103) 0:01:55.713 ********* 2021-03-09T09:35:01.109 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T09:35:01.111 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.133 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.104) 0:01:55.817 ********* 2021-03-09T09:35:01.202 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T09:35:01.204 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.235 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.101) 0:01:55.919 ********* 2021-03-09T09:35:01.243 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-09T09:35:01.244 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:31:49 UTC", "ActiveEnterTimestampMonotonic": "17852460", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target sysinit.target nss-lookup.target system.slice systemd-journald.socket basic.target -.mount systemd-tmpfiles-setup.service network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:31:49 UTC", "AssertTimestampMonotonic": "17472680", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:31:49 UTC", "ConditionTimestampMonotonic": "17472680", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-09T09:35:01.246 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1460", "ExecMainStartTimestamp": "Tue 2021-03-09 09:31:49 UTC", "ExecMainStartTimestampMonotonic": "17852447", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:31:49 UTC", "InactiveExitTimestampMonotonic": "17473791", "InvocationID": "61559bb60ae74778962311a6b512b6a2", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-09T09:35:01.247 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1460", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-09T09:35:01.249 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:31:49 UTC", "StateChangeTimestampMonotonic": "17852460", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:31:49 UTC", "WatchdogTimestampMonotonic": "17852459", "WatchdogUSec": "0"}} 2021-03-09T09:35:01.274 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.038) 0:01:55.958 ********* 2021-03-09T09:35:01.303 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T09:35:01.306 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.328 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.054) 0:01:56.012 ********* 2021-03-09T09:35:01.346 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T09:35:01.348 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.371 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.042) 0:01:56.055 ********* 2021-03-09T09:35:01.411 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T09:35:01.413 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.437 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.065) 0:01:56.121 ********* 2021-03-09T09:35:01.453 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T09:35:01.455 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.480 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.043) 0:01:56.164 ********* 2021-03-09T09:35:01.508 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.534 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.053) 0:01:56.218 ********* 2021-03-09T09:35:01.551 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T09:35:01.553 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.578 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.044) 0:01:56.262 ********* 2021-03-09T09:35:01.607 INFO:teuthology.task.ansible.out: 2021-03-09T09:35:01.609 INFO:teuthology.task.ansible.out:TASK [testnode : Remove packages] ********************************************** skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.633 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.054) 0:01:56.317 ********* 2021-03-09T09:35:01.661 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.685 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.052) 0:01:56.369 ********* 2021-03-09T09:35:01.739 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T09:35:01.742 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.767 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.081) 0:01:56.451 ********* 2021-03-09T09:35:01.785 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T09:35:01.787 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.816 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.049) 0:01:56.500 ********* 2021-03-09T09:35:01.848 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T09:35:01.850 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.877 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.060) 0:01:56.561 ********* 2021-03-09T09:35:01.895 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T09:35:01.897 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.921 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.044) 0:01:56.605 ********* 2021-03-09T09:35:01.950 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T09:35:01.952 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:01.975 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:01 +0000 (0:00:00.053) 0:01:56.659 ********* 2021-03-09T09:35:02.003 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T09:35:02.005 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.029 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.053) 0:01:56.713 ********* 2021-03-09T09:35:02.047 INFO:teuthology.task.ansible.out: 2021-03-09T09:35:02.049 INFO:teuthology.task.ansible.out:TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.076 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.046) 0:01:56.760 ********* 2021-03-09T09:35:02.104 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T09:35:02.106 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.129 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.053) 0:01:56.813 ********* 2021-03-09T09:35:02.147 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:35:02.149 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.173 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.043) 0:01:56.857 ********* 2021-03-09T09:35:02.213 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:02.215 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.239 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.065) 0:01:56.923 ********* 2021-03-09T09:35:02.258 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:35:02.260 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.296 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.057) 0:01:56.980 ********* 2021-03-09T09:35:02.325 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T09:35:02.327 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.352 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.055) 0:01:57.036 ********* 2021-03-09T09:35:02.371 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T09:35:02.373 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.398 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.045) 0:01:57.082 ********* 2021-03-09T09:35:02.438 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T09:35:02.440 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.469 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.071) 0:01:57.153 ********* 2021-03-09T09:35:02.477 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T09:35:02.479 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.503 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.034) 0:01:57.187 ********* 2021-03-09T09:35:02.554 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T09:35:02.556 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.601 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.097) 0:01:57.285 ********* 2021-03-09T09:35:02.611 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T09:35:02.613 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.641 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.040) 0:01:57.325 ********* 2021-03-09T09:35:02.685 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T09:35:02.687 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.716 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.074) 0:01:57.400 ********* 2021-03-09T09:35:02.724 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T09:35:02.726 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.751 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.035) 0:01:57.435 ********* 2021-03-09T09:35:02.791 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T09:35:02.793 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.818 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.066) 0:01:57.502 ********* 2021-03-09T09:35:02.835 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:35:02.837 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.861 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.043) 0:01:57.546 ********* 2021-03-09T09:35:02.890 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:35:02.892 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.923 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.060) 0:01:57.606 ********* 2021-03-09T09:35:02.951 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:35:02.953 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:02.985 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:02 +0000 (0:00:00.062) 0:01:57.669 ********* 2021-03-09T09:35:03.006 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:35:03.007 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.032 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.047) 0:01:57.716 ********* 2021-03-09T09:35:03.072 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T09:35:03.074 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.106 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.073) 0:01:57.790 ********* 2021-03-09T09:35:03.123 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T09:35:03.125 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.153 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.047) 0:01:57.837 ********* 2021-03-09T09:35:03.183 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:35:03.185 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.209 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.055) 0:01:57.893 ********* 2021-03-09T09:35:03.236 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T09:35:03.238 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.264 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.055) 0:01:57.948 ********* 2021-03-09T09:35:03.285 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:35:03.286 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.310 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.045) 0:01:57.994 ********* 2021-03-09T09:35:03.339 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T09:35:03.341 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.365 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.054) 0:01:58.049 ********* 2021-03-09T09:35:03.382 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-09T09:35:03.384 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.408 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.043) 0:01:58.092 ********* 2021-03-09T09:35:03.436 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T09:35:03.438 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.465 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.056) 0:01:58.149 ********* 2021-03-09T09:35:03.550 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:35:03.552 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.576 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.111) 0:01:58.260 ********* 2021-03-09T09:35:03.625 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-09T09:35:03.627 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-09T09:35:03.653 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.076) 0:01:58.337 ********* 2021-03-09T09:35:03.672 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:35:03.673 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.701 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.048) 0:01:58.385 ********* 2021-03-09T09:35:03.799 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T09:35:03.801 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.829 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.127) 0:01:58.513 ********* 2021-03-09T09:35:03.910 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:35:03.911 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:03.935 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:03 +0000 (0:00:00.106) 0:01:58.619 ********* 2021-03-09T09:35:04.004 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:35:04.006 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:04.034 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:04 +0000 (0:00:00.099) 0:01:58.718 ********* 2021-03-09T09:35:04.074 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T09:35:04.076 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-09T09:35:04.100 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:04 +0000 (0:00:00.065) 0:01:58.784 ********* 2021-03-09T09:35:04.118 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-09T09:35:04.120 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:04.148 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:04 +0000 (0:00:00.048) 0:01:58.832 ********* 2021-03-09T09:35:04.168 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T09:35:04.169 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:04.194 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:04 +0000 (0:00:00.046) 0:01:58.879 ********* 2021-03-09T09:35:04.349 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-09T09:35:04.351 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-09T09:35:04.374 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:04 +0000 (0:00:00.179) 0:01:59.058 ********* 2021-03-09T09:35:04.611 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-09T09:35:04.613 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-09T09:35:04.635 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:04 +0000 (0:00:00.261) 0:01:59.319 ********* 2021-03-09T09:35:04.789 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T09:35:04.790 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-09T09:35:04.814 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:04 +0000 (0:00:00.178) 0:01:59.498 ********* 2021-03-09T09:35:04.905 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T09:35:04.907 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:04.940 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:04 +0000 (0:00:00.125) 0:01:59.624 ********* 2021-03-09T09:35:05.033 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-09T09:35:05.035 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-09T09:35:05.072 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:05 +0000 (0:00:00.132) 0:01:59.756 ********* 2021-03-09T09:35:05.347 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-09T09:35:05.349 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-09T09:35:05.384 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:05 +0000 (0:00:00.312) 0:02:00.068 ********* 2021-03-09T09:35:05.467 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-09T09:35:05.469 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-09T09:35:05.496 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:05 +0000 (0:00:00.111) 0:02:00.180 ********* 2021-03-09T09:35:05.682 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-09T09:35:05.684 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:35:05.708 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:05 +0000 (0:00:00.212) 0:02:00.392 ********* 2021-03-09T09:35:05.798 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-09T09:35:05.800 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-09T09:35:05.825 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:05 +0000 (0:00:00.117) 0:02:00.509 ********* 2021-03-09T09:35:05.850 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:05 +0000 (0:00:00.024) 0:02:00.534 ********* 2021-03-09T09:35:05.920 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:05.943 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:05 +0000 (0:00:00.093) 0:02:00.627 ********* 2021-03-09T09:35:06.011 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-09T09:35:06.013 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.043 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.099) 0:02:00.726 ********* 2021-03-09T09:35:06.116 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-09T09:35:06.117 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.140 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.098) 0:02:00.825 ********* 2021-03-09T09:35:06.203 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-09T09:35:06.204 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-09T09:35:06.234 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.093) 0:02:00.918 ********* 2021-03-09T09:35:06.242 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-09T09:35:06.244 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.271 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.037) 0:02:00.956 ********* 2021-03-09T09:35:06.362 INFO:teuthology.task.ansible.out: 2021-03-09T09:35:06.364 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-09T09:35:06.392 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.120) 0:02:01.076 ********* 2021-03-09T09:35:06.420 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-09T09:35:06.422 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:35:06.445 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.052) 0:02:01.129 ********* 2021-03-09T09:35:06.494 INFO:teuthology.task.ansible.out: 2021-03-09T09:35:06.496 INFO:teuthology.task.ansible.out:TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi194.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.506 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.508 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.518 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.520 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.543 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.098) 0:02:01.227 ********* 2021-03-09T09:35:06.551 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-09T09:35:06.554 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [smithi194.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi194.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.554 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.566 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.568 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.580 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.590 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.637 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.093) 0:02:01.321 ********* 2021-03-09T09:35:06.643 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-09T09:35:06.645 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.671 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.034) 0:02:01.355 ********* 2021-03-09T09:35:06.751 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-09T09:35:06.753 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.779 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.108) 0:02:01.463 ********* 2021-03-09T09:35:06.860 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-09T09:35:06.862 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.885 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.105) 0:02:01.569 ********* 2021-03-09T09:35:06.967 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-09T09:35:06.969 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:06.992 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:06 +0000 (0:00:00.106) 0:02:01.676 ********* 2021-03-09T09:35:07.009 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-09T09:35:07.010 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.009383", "end": "2021-03-09 09:35:06.977552", "rc": 0, "start": "2021-03-09 09:35:06.968169", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-09T09:35:07.038 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:07 +0000 (0:00:00.046) 0:02:01.722 ********* 2021-03-09T09:35:07.077 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:07.079 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-09T09:35:07.104 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:07 +0000 (0:00:00.065) 0:02:01.788 ********* 2021-03-09T09:35:07.122 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-09T09:35:07.124 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.150 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:07 +0000 (0:00:00.046) 0:02:01.834 ********* 2021-03-09T09:35:07.189 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-09T09:35:07.191 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.201 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.201 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.212 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.223 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.225 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.257 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:07 +0000 (0:00:00.107) 0:02:01.941 ********* 2021-03-09T09:35:07.266 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-09T09:35:07.268 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.279 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.281 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.291 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.293 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.303 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.334 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:07 +0000 (0:00:00.076) 0:02:02.018 ********* 2021-03-09T09:35:07.342 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-09T09:35:07.344 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.374 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:07 +0000 (0:00:00.040) 0:02:02.058 ********* 2021-03-09T09:35:07.613 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-09T09:35:07.615 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003092", "end": "2021-03-09 09:35:07.588184", "rc": 0, "start": "2021-03-09 09:35:07.585092", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-09T09:35:07.654 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:07 +0000 (0:00:00.279) 0:02:02.338 ********* 2021-03-09T09:35:07.724 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-09T09:35:07.726 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.755 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:07 +0000 (0:00:00.101) 0:02:02.439 ********* 2021-03-09T09:35:07.836 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-09T09:35:07.838 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.859 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:07 +0000 (0:00:00.104) 0:02:02.544 ********* 2021-03-09T09:35:07.939 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-09T09:35:07.941 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:07.962 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:07 +0000 (0:00:00.102) 0:02:02.646 ********* 2021-03-09T09:35:08.344 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-09T09:35:08.346 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:35:08.368 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:08 +0000 (0:00:00.405) 0:02:03.052 ********* 2021-03-09T09:35:08.447 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-09T09:35:08.449 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:08.472 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:08 +0000 (0:00:00.103) 0:02:03.156 ********* 2021-03-09T09:35:08.562 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T09:35:08.564 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-09T09:35:08.586 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:08 +0000 (0:00:00.114) 0:02:03.271 ********* 2021-03-09T09:35:08.781 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-09T09:35:08.783 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009830", "end": "2021-03-09 09:35:08.723675", "rc": 0, "start": "2021-03-09 09:35:08.713845", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:08.805 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:08 +0000 (0:00:00.218) 0:02:03.489 ********* 2021-03-09T09:35:08.885 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-09T09:35:08.887 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:08.909 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:08 +0000 (0:00:00.103) 0:02:03.593 ********* 2021-03-09T09:35:08.926 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-09T09:35:08.927 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:35:08.951 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:08 +0000 (0:00:00.041) 0:02:03.635 ********* 2021-03-09T09:35:08.978 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-09T09:35:09.002 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:09 +0000 (0:00:00.051) 0:02:03.686 ********* 2021-03-09T09:35:09.020 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-09T09:35:09.021 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:09.047 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:09 +0000 (0:00:00.045) 0:02:03.732 ********* 2021-03-09T09:35:09.106 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:35:09.108 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:09.129 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T09:35:09.130 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'block_used': 13645814, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226449294, 'size_available': 927536308224, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226449294, "block_size": 4096, "block_total": 240095108, "block_used": 13645814, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927536308224, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-09T09:35:09.154 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:09 +0000 (0:00:00.106) 0:02:03.838 ********* 2021-03-09T09:35:09.338 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-09T09:35:09.340 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.008154", "end": "2021-03-09 09:35:09.322394", "rc": 0, "start": "2021-03-09 09:35:09.314240", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:09.363 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:09 +0000 (0:00:00.208) 0:02:04.047 ********* 2021-03-09T09:35:09.452 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-09T09:35:09.454 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'block_used': 13645814, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226449294, 'size_available': 927536308224, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226449294, "block_size": 4096, "block_total": 240095108, "block_used": 13645814, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927536308224, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:09.477 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:09 +0000 (0:00:00.114) 0:02:04.161 ********* 2021-03-09T09:35:09.546 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-09T09:35:09.569 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:09 +0000 (0:00:00.091) 0:02:04.253 ********* 2021-03-09T09:35:09.659 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:35:09.660 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:10.833 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWae22vq5nDw0M2vJ2dA0UiyGclV5fbKd3'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.599643", "end": "2021-03-09 09:35:10.764499", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWae22vq5nDw0M2vJ2dA0UiyGclV5fbKd3"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:09.164856", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:10.844 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:10.845 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:11.453 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NNht8wZY8nul0Jzxoa4Az3XNuyvLTWLYS'], 'uuids': ['eaff056d-dd9b-4523-b661-d8b02968a6f6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.654509", "end": "2021-03-09 09:35:11.418191", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NNht8wZY8nul0Jzxoa4Az3XNuyvLTWLYS"], "labels": [], "masters": [], "uuids": ["eaff056d-dd9b-4523-b661-d8b02968a6f6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:09.763682", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:11.455 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:11.469 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:12.089 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsW6MMEzUDtm80ifW95S9Du2eeccM9qYE3J'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.107420", "end": "2021-03-09 09:35:12.021541", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsW6MMEzUDtm80ifW95S9Du2eeccM9qYE3J"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:10.914121", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:12.095 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNHZ', 'wwn-0x5000c50092941eb0'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNHZ-part1', 'wwn-0x5000c50092941eb0-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092941eb0', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XNHZ', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNHZ", "wwn-0x5000c50092941eb0"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNHZ-part1", "wwn-0x5000c50092941eb0-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNHZ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092941eb0"}}, "skip_reason": 2021-03-09T09:35:12.097 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-09T09:35:12.703 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18N6xk2x7Vhx47jweDdcZF08k9B8E34Hg3F'], 'uuids': ['26b04989-27a1-4543-bd9c-56b6f86e0ade']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.097812", "end": "2021-03-09 09:35:12.674739", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18N6xk2x7Vhx47jweDdcZF08k9B8E34Hg3F"], "labels": [], "masters": [], "uuids": ["26b04989-27a1-4543-bd9c-56b6f86e0ade"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:11.576927", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:12.705 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS', 'wwn-0x5000c500871a48d9'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1', 'wwn-0x5000c500871a48d9-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a48d9', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HRYS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS", "wwn-0x5000c500871a48d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1", "wwn-0x5000c500871a48d9-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HRYS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a48d9"}}, "skip_reason": 2021-03-09T09:35:12.707 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-09T09:35:13.407 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-5coqhV-JxYc-ZqWh-cOIz-oCkT-0CiQ-NRylg4', 'nvme-INTEL_SSDPEDMD400G4_CVFT6240002A400BGN', 'nvme-nvme.8086-43564654363234303030324134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.170158", "end": "2021-03-09 09:35:13.336868", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-5coqhV-JxYc-ZqWh-cOIz-oCkT-0CiQ-NRylg4", "nvme-INTEL_SSDPEDMD400G4_CVFT6240002A400BGN", "nvme-nvme.8086-43564654363234303030324134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:12.166710", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-09T09:35:13.409 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:13.409 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:13.421 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:13.959 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-qUV2CY-dXDI-Ictf-pnM4-L2ay-eZLn-XVW9jt', 'nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN', 'nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.114354", "end": "2021-03-09 09:35:13.926592", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-qUV2CY-dXDI-Ictf-pnM4-L2ay-eZLn-XVW9jt", "nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN", "nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:12.812238", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-03-09T09:35:13.961 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:13.971 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:13.972 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:14.861 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWb79cm5VTnMGTAlyC9k03Dk1lCJ0gqeiO'], 'uuids': ['3be98854-b976-4bad-a651-3980578a8d71']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.319677", "end": "2021-03-09 09:35:14.806089", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWb79cm5VTnMGTAlyC9k03Dk1lCJ0gqeiO"], "labels": [], "masters": [], "uuids": ["3be98854-b976-4bad-a651-3980578a8d71"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:13.486412", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:14.874 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:14.885 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:15.211 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18N0SNYcHet3BeOhiPs0x1gzZEsdbLyZ5KV'], 'uuids': ['022a5767-1be1-477a-883e-48eeca99fe44']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.106018", "end": "2021-03-09 09:35:15.182440", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18N0SNYcHet3BeOhiPs0x1gzZEsdbLyZ5KV"], "labels": [], "masters": [], "uuids": ["022a5767-1be1-477a-883e-48eeca99fe44"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:14.076422", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:15.223 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:15.224 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:16.348 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWNDdHVi5RdV1tRAu22k1tK8mpFaxsp4to'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.333624", "end": "2021-03-09 09:35:16.282722", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWNDdHVi5RdV1tRAu22k1tK8mpFaxsp4to"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:14.949098", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:16.359 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:16.465 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NzV9BIcyH6w7H0woanyNpJWTX4NdQoWeI'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.103793", "end": "2021-03-09 09:35:16.433414", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NzV9BIcyH6w7H0woanyNpJWTX4NdQoWeI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:15.329621", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:16.466 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:17.701 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NxyoLrCOTjHdLt6NrzXBlh2tzkBxxBqqw'], 'uuids': ['42d3e9eb-1822-4e9e-97b8-c41694e021a2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.100409", "end": "2021-03-09 09:35:17.674619", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NxyoLrCOTjHdLt6NrzXBlh2tzkBxxBqqw"], "labels": [], "masters": [], "uuids": ["42d3e9eb-1822-4e9e-97b8-c41694e021a2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:16.574210", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:17.757 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:17 +0000 (0:00:08.188) 0:02:12.441 ********* 2021-03-09T09:35:17.766 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWYwB4N1zxWEu1CxXnd9krU6vCHal0R6DB'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.276440", "end": "2021-03-09 09:35:17.707718", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWYwB4N1zxWEu1CxXnd9krU6vCHal0R6DB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:16.431278", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:35:17.815 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:17 +0000 (0:00:00.057) 0:02:12.499 ********* 2021-03-09T09:35:17.909 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T09:35:17.911 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.016 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.206 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWae22vq5nDw0M2vJ2dA0UiyGclV5fbKd3'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.155725", "end": "2021-03-09 09:35:18.143365", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWae22vq5nDw0M2vJ2dA0UiyGclV5fbKd3"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:17.987640", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150929 s, 764 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150929 s, 764 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:18.218 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.230 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.252 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NNht8wZY8nul0Jzxoa4Az3XNuyvLTWLYS'], 'uuids': ['eaff056d-dd9b-4523-b661-d8b02968a6f6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.114441", "end": "2021-03-09 09:35:18.230307", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NNht8wZY8nul0Jzxoa4Az3XNuyvLTWLYS"], "labels": [], "masters": [], "uuids": ["eaff056d-dd9b-4523-b661-d8b02968a6f6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:18.115866", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.11143 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.11143 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:18.254 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.266 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.496 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18N6xk2x7Vhx47jweDdcZF08k9B8E34Hg3F'], 'uuids': ['26b04989-27a1-4543-bd9c-56b6f86e0ade']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.109902", "end": "2021-03-09 09:35:18.475771", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18N6xk2x7Vhx47jweDdcZF08k9B8E34Hg3F"], "labels": [], "masters": [], "uuids": ["26b04989-27a1-4543-bd9c-56b6f86e0ade"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:18.365869", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.105644 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.105644 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:18.508 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS', 'wwn-0x5000c500871a48d9'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1', 'wwn-0x5000c500871a48d9-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a48d9', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HRYS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS", "wwn-0x5000c500871a48d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1", "wwn-0x5000c500871a48d9-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HRYS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a48d9"}}, "skip_reason": 2021-03-09T09:35:18.510 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-09T09:35:18.519 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsW6MMEzUDtm80ifW95S9Du2eeccM9qYE3J'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.159900", "end": "2021-03-09 09:35:18.450173", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsW6MMEzUDtm80ifW95S9Du2eeccM9qYE3J"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:18.290273", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155226 s, 743 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155226 s, 743 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:18.521 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNHZ', 'wwn-0x5000c50092941eb0'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNHZ-part1', 'wwn-0x5000c50092941eb0-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092941eb0', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XNHZ', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNHZ", "wwn-0x5000c50092941eb0"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNHZ-part1", "wwn-0x5000c50092941eb0-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNHZ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092941eb0"}}, "skip_reason": 2021-03-09T09:35:18.523 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-09T09:35:18.691 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-qUV2CY-dXDI-Ictf-pnM4-L2ay-eZLn-XVW9jt', 'nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN', 'nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.051657", "end": "2021-03-09 09:35:18.657795", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-qUV2CY-dXDI-Ictf-pnM4-L2ay-eZLn-XVW9jt", "nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN", "nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:18.606138", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0468861 s, 2.5 GB/s", "stderr_lines": [" 2021-03-09T09:35:18.693 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0468861 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:18.702 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.703 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-5coqhV-JxYc-ZqWh-cOIz-oCkT-0CiQ-NRylg4', 'nvme-INTEL_SSDPEDMD400G4_CVFT6240002A400BGN', 'nvme-nvme.8086-43564654363234303030324134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.051711", "end": "2021-03-09 09:35:18.639604", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-5coqhV-JxYc-ZqWh-cOIz-oCkT-0CiQ-NRylg4", "nvme-INTEL_SSDPEDMD400G4_CVFT6240002A400BGN", "nvme-nvme.8086-43564654363234303030324134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:18.587893", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0480213 s, 2.4 GB/s", "std 2021-03-09T09:35:18.705 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0480213 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.706 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.717 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.929 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18N0SNYcHet3BeOhiPs0x1gzZEsdbLyZ5KV'], 'uuids': ['022a5767-1be1-477a-883e-48eeca99fe44']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.107347", "end": "2021-03-09 09:35:18.907180", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18N0SNYcHet3BeOhiPs0x1gzZEsdbLyZ5KV"], "labels": [], "masters": [], "uuids": ["022a5767-1be1-477a-883e-48eeca99fe44"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:18.799833", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.103712 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.103712 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:18.941 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:18.953 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:19.152 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWb79cm5VTnMGTAlyC9k03Dk1lCJ0gqeiO'], 'uuids': ['3be98854-b976-4bad-a651-3980578a8d71']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.156827", "end": "2021-03-09 09:35:19.081886", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWb79cm5VTnMGTAlyC9k03Dk1lCJ0gqeiO"], "labels": [], "masters": [], "uuids": ["3be98854-b976-4bad-a651-3980578a8d71"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:18.925059", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15204 s, 759 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15204 s, 759 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:19.153 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:19.165 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:19.187 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NzV9BIcyH6w7H0woanyNpJWTX4NdQoWeI'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.109419", "end": "2021-03-09 09:35:19.161749", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NzV9BIcyH6w7H0woanyNpJWTX4NdQoWeI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:19.052330", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.106344 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.106344 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:19.200 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:19.440 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NxyoLrCOTjHdLt6NrzXBlh2tzkBxxBqqw'], 'uuids': ['42d3e9eb-1822-4e9e-97b8-c41694e021a2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.107754", "end": "2021-03-09 09:35:19.415320", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-alM4oLWdVFblOxAvGLa3b7kCY7wXM18NxyoLrCOTjHdLt6NrzXBlh2tzkBxxBqqw"], "labels": [], "masters": [], "uuids": ["42d3e9eb-1822-4e9e-97b8-c41694e021a2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:19.307566", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.103057 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.103057 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:19.477 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:19 +0000 (0:00:01.662) 0:02:14.161 ********* 2021-03-09T09:35:19.487 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWNDdHVi5RdV1tRAu22k1tK8mpFaxsp4to'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.168062", "end": "2021-03-09 09:35:19.400482", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWNDdHVi5RdV1tRAu22k1tK8mpFaxsp4to"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:19.232420", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.163149 s, 707 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.163149 s, 707 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:19.488 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:19.754 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWYwB4N1zxWEu1CxXnd9krU6vCHal0R6DB'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.154678", "end": "2021-03-09 09:35:19.687361", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-j5IB7M68Jj1aPPKfUwwsc9JYNLvCYqsWYwB4N1zxWEu1CxXnd9krU6vCHal0R6DB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:35:19.532683", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151534 s, 761 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151534 s, 761 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:19.811 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:19 +0000 (0:00:00.334) 0:02:14.495 ********* 2021-03-09T09:35:19.946 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-09T09:35:19.948 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.147232", "end": "2021-03-09 09:35:19.738658", "rc": 0, "start": "2021-03-09 09:35:19.591426", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:19.973 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:19 +0000 (0:00:00.161) 0:02:14.657 ********* 2021-03-09T09:35:20.179 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-09T09:35:20.181 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.071784", "end": "2021-03-09 09:35:20.161057", "rc": 0, "start": "2021-03-09 09:35:20.089273", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:20.203 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.230) 0:02:14.887 ********* 2021-03-09T09:35:20.263 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-09T09:35:20.324 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.120) 0:02:15.008 ********* 2021-03-09T09:35:20.331 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-09T09:35:20.333 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.157350", "end": "2021-03-09 09:35:20.056562", "rc": 0, "start": "2021-03-09 09:35:19.899212", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:20.360 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.035) 0:02:15.044 ********* 2021-03-09T09:35:20.366 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-09T09:35:20.368 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:20.396 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.035) 0:02:15.080 ********* 2021-03-09T09:35:20.424 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-09T09:35:20.426 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:20.452 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.056) 0:02:15.136 ********* 2021-03-09T09:35:20.479 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T09:35:20.481 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:20.636 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.184) 0:02:15.320 ********* 2021-03-09T09:35:20.807 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.054467", "end": "2021-03-09 09:35:20.490698", "rc": 0, "start": "2021-03-09 09:35:20.436231", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.038) 0:02:15.359 ********* TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.033) 0:02:15.392 ********* TASK [testnode : Remove physical volumes] ************************************** Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.040) 0:02:15.433 ********* TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:20.809 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.059) 0:02:15.492 ********* 2021-03-09T09:35:20.847 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-09T09:35:20.850 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:35:20.873 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.064) 0:02:15.557 ********* 2021-03-09T09:35:20.895 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.021) 0:02:15.579 ********* 2021-03-09T09:35:20.912 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T09:35:20.914 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'block_used': 13645814, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226449294, 'size_available': 927536308224, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226449294, "block_size": 4096, "block_total": 240095108, "block_used": 13645814, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927536308224, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:20.937 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.042) 0:02:15.621 ********* 2021-03-09T09:35:20.944 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T09:35:20.946 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:20.973 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:20 +0000 (0:00:00.035) 0:02:15.657 ********* 2021-03-09T09:35:20.979 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.005 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.032) 0:02:15.689 ********* 2021-03-09T09:35:21.013 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-09T09:35:21.014 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.039 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.034) 0:02:15.723 ********* 2021-03-09T09:35:21.045 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:21.047 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.073 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.034) 0:02:15.757 ********* 2021-03-09T09:35:21.113 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:21.115 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.146 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.072) 0:02:15.830 ********* 2021-03-09T09:35:21.175 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-09T09:35:21.177 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.200 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.054) 0:02:15.884 ********* 2021-03-09T09:35:21.224 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.023) 0:02:15.908 ********* 2021-03-09T09:35:21.232 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-09T09:35:21.234 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.271 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.046) 0:02:15.955 ********* 2021-03-09T09:35:21.278 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T09:35:21.279 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.309 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.038) 0:02:15.993 ********* 2021-03-09T09:35:21.318 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-09T09:35:21.320 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.345 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.035) 0:02:16.029 ********* 2021-03-09T09:35:21.352 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-09T09:35:21.353 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.379 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.034) 0:02:16.063 ********* 2021-03-09T09:35:21.419 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:21.421 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.443 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.063) 0:02:16.127 ********* 2021-03-09T09:35:21.471 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:21.473 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.501 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.058) 0:02:16.185 ********* 2021-03-09T09:35:21.528 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-09T09:35:21.530 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.553 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.052) 0:02:16.237 ********* 2021-03-09T09:35:21.581 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-09T09:35:21.583 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.605 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.051) 0:02:16.289 ********* 2021-03-09T09:35:21.632 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-09T09:35:21.634 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:21.656 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.051) 0:02:16.340 ********* 2021-03-09T09:35:21.850 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-09T09:35:21.852 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-09T09:35:21.885 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:21 +0000 (0:00:00.228) 0:02:16.569 ********* 2021-03-09T09:35:21.979 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-09T09:35:22.016 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:22 +0000 (0:00:00.130) 0:02:16.700 ********* 2021-03-09T09:35:22.379 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T09:35:22.381 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:35:22.391 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:35:22.619 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:35:22.662 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:35:22.859 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:35:22.882 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:35:23.112 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:35:23.114 INFO:teuthology.task.ansible.out: 2021-03-09T09:35:23.116 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:35:23.356 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:35:23.380 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:23 +0000 (0:00:01.364) 0:02:18.064 ********* 2021-03-09T09:35:23.387 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:35:23.415 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:23 +0000 (0:00:00.034) 0:02:18.099 ********* 2021-03-09T09:35:23.526 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T09:35:23.528 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-09T09:35:23.561 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:23 +0000 (0:00:00.145) 0:02:18.245 ********* 2021-03-09T09:35:23.568 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-09T09:35:23.594 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:23 +0000 (0:00:00.033) 0:02:18.278 ********* 2021-03-09T09:35:23.716 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T09:35:23.717 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:35:23.780 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:35:23.833 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:35:23.902 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:35:23.904 INFO:teuthology.task.ansible.out: 2021-03-09T09:35:23.946 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:35:24.020 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:35:24.063 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:35:24.085 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:24.115 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.520) 0:02:18.799 ********* 2021-03-09T09:35:24.138 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:35:24.138 INFO:teuthology.task.ansible.out: 2021-03-09T09:35:24.140 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:24.167 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.052) 0:02:18.851 ********* 2021-03-09T09:35:24.173 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-09T09:35:24.175 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:24.198 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.031) 0:02:18.882 ********* 2021-03-09T09:35:24.205 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:24.229 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.031) 0:02:18.913 ********* 2021-03-09T09:35:24.236 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-09T09:35:24.238 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:24.261 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.032) 0:02:18.946 ********* 2021-03-09T09:35:24.268 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:24.293 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.031) 0:02:18.977 ********* 2021-03-09T09:35:24.405 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-09T09:35:24.407 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:35:24.431 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.137) 0:02:19.115 ********* 2021-03-09T09:35:24.439 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:35:24.464 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.033) 0:02:19.148 ********* 2021-03-09T09:35:24.470 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-09T09:35:24.472 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-09T09:35:24.498 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.034) 0:02:19.182 ********* 2021-03-09T09:35:24.504 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-09T09:35:24.530 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.031) 0:02:19.214 ********* 2021-03-09T09:35:24.817 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-03-09T09:35:24.819 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true} 2021-03-09T09:35:24.850 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.319) 0:02:19.534 ********* 2021-03-09T09:35:24.889 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true} 2021-03-09T09:35:24.914 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:24 +0000 (0:00:00.063) 0:02:19.598 ********* 2021-03-09T09:35:25.195 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-09T09:35:25.197 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-09T09:35:25.219 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.304) 0:02:19.903 ********* 2021-03-09T09:35:25.225 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-09T09:35:25.250 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.030) 0:02:19.934 ********* 2021-03-09T09:35:25.256 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:25.281 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.030) 0:02:19.964 ********* 2021-03-09T09:35:25.286 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:25.311 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.030) 0:02:19.995 ********* 2021-03-09T09:35:25.318 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-09T09:35:25.319 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:25.342 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.031) 0:02:20.027 ********* 2021-03-09T09:35:25.349 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:25.375 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.032) 0:02:20.059 ********* 2021-03-09T09:35:25.382 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T09:35:25.384 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:25.408 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.033) 0:02:20.092 ********* 2021-03-09T09:35:25.415 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:25.443 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.034) 0:02:20.127 ********* 2021-03-09T09:35:25.450 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T09:35:25.452 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:25.475 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.032) 0:02:20.159 ********* 2021-03-09T09:35:25.483 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:25.518 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.043) 0:02:20.202 ********* 2021-03-09T09:35:25.656 INFO:teuthology.task.ansible.out: 2021-03-09T09:35:25.657 INFO:teuthology.task.ansible.out:TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T09:35:25.682 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.163) 0:02:20.366 ********* 2021-03-09T09:35:25.704 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T09:35:25.734 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.052) 0:02:20.418 ********* 2021-03-09T09:35:25.828 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-09T09:35:25.830 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:35:25.856 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.121) 0:02:20.540 ********* 2021-03-09T09:35:25.898 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:35:25.924 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:25 +0000 (0:00:00.067) 0:02:20.608 ********* 2021-03-09T09:35:26.253 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-09T09:35:26.255 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-09T09:35:26.278 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:26 +0000 (0:00:00.353) 0:02:20.962 ********* 2021-03-09T09:35:26.296 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-09T09:35:26.298 INFO:teuthology.task.ansible.out: 2021-03-09T09:35:26.330 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:26 +0000 (0:00:00.051) 0:02:21.014 ********* 2021-03-09T09:35:26.658 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-09T09:35:26.660 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-09T09:35:26.685 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:26 +0000 (0:00:00.355) 0:02:21.369 ********* 2021-03-09T09:35:26.703 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-09T09:35:26.731 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:26 +0000 (0:00:00.045) 0:02:21.415 ********* 2021-03-09T09:35:27.597 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-03-09T09:35:27.599 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:35:27.622 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:27 +0000 (0:00:00.891) 0:02:22.306 ********* 2021-03-09T09:35:27.673 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:35:27.698 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:27 +0000 (0:00:00.075) 0:02:22.382 ********* 2021-03-09T09:35:27.809 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-09T09:35:27.811 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047540", "end": "2021-03-09 09:35:27.751099", "rc": 0, "start": "2021-03-09 09:35:27.703559", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-03-09T09:35:27.834 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:27 +0000 (0:00:00.136) 0:02:22.518 ********* 2021-03-09T09:35:27.861 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-03-09T09:35:27.863 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:27.886 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:27 +0000 (0:00:00.051) 0:02:22.570 ********* 2021-03-09T09:35:27.893 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-09T09:35:27.895 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.045904", "end": "2021-03-09 09:35:27.858803", "rc": 0, "start": "2021-03-09 09:35:27.812899", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-03-09T09:35:27.925 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:27 +0000 (0:00:00.039) 0:02:22.609 ********* 2021-03-09T09:35:27.932 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-09T09:35:27.934 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:27.958 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:27 +0000 (0:00:00.032) 0:02:22.642 ********* 2021-03-09T09:35:27.964 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-03-09T09:35:27.966 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:27.991 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:27 +0000 (0:00:00.033) 0:02:22.675 ********* 2021-03-09T09:35:28.020 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-09T09:35:28.021 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:28.044 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:28 +0000 (0:00:00.052) 0:02:22.728 ********* 2021-03-09T09:35:28.796 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-09T09:35:28.798 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:35:28.821 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:28 +0000 (0:00:00.777) 0:02:23.505 ********* 2021-03-09T09:35:28.848 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-09T09:35:28.850 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:28.878 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:28 +0000 (0:00:00.056) 0:02:23.562 ********* 2021-03-09T09:35:28.978 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-09T09:35:28.980 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:35:29.005 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.126) 0:02:23.689 ********* 2021-03-09T09:35:29.033 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-09T09:35:29.034 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:29.058 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.053) 0:02:23.742 ********* 2021-03-09T09:35:29.210 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-09T09:35:29.212 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-09T09:35:29.237 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.179) 0:02:23.921 ********* 2021-03-09T09:35:29.264 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-09T09:35:29.266 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:29.289 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.051) 0:02:23.973 ********* 2021-03-09T09:35:29.399 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-09T09:35:29.401 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-09T09:35:29.425 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.136) 0:02:24.109 ********* 2021-03-09T09:35:29.452 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-09T09:35:29.454 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:29.476 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.050) 0:02:24.160 ********* 2021-03-09T09:35:29.505 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-09T09:35:29.506 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:32:15 UTC", "ActiveEnterTimestampMonotonic": "16916242", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice -.mount systemd-tmpfiles-setup.service basic.target systemd-journald.socket network.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:32:15 UTC", "AssertTimestampMonotonic": "16880113", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:32:15 UTC", "ConditionTimestampMonotonic": "16880113", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-09T09:35:29.508 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1079", "ExecMainStartTimestamp": "Tue 2021-03-09 09:32:15 UTC", "ExecMainStartTimestampMonotonic": "16916230", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:32:15 UTC", "InactiveExitTimestampMonotonic": "16881010", "InvocationID": "1e88cbed3a7b4bbe9b435b847cee06cb", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-09T09:35:29.509 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1079", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:32:15 UTC", "S 2021-03-09T09:35:29.511 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "16916242", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:32:15 UTC", "WatchdogTimestampMonotonic": "16916241", "WatchdogUSec": "0"}} 2021-03-09T09:35:29.533 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.057) 0:02:24.217 ********* 2021-03-09T09:35:29.561 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:29.562 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-09T09:35:29.584 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.051) 0:02:24.268 ********* 2021-03-09T09:35:29.611 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:29.613 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:29.636 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.051) 0:02:24.320 ********* 2021-03-09T09:35:29.663 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-09T09:35:29.665 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:29.687 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.051) 0:02:24.371 ********* 2021-03-09T09:35:29.704 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-09T09:35:29.706 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:31:54 UTC", "ActiveEnterTimestampMonotonic": "23560162", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target systemd-journald.socket sysinit.target system.slice systemd-tmpfiles-setup.service basic.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:31:48 UTC", "AssertTimestampMonotonic": "16984113", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:31:48 UTC", "ConditionTimestampMonotonic": "16984113", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-09T09:35:29.707 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2040", "ExecMainStartTimestamp": "Tue 2021-03-09 09:31:54 UTC", "ExecMainStartTimestampMonotonic": "23560149", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:31:48 UTC", "InactiveExitTimestampMonotonic": "16984937", "InvocationID": "933a1166a05c4e369369d5acce3d296a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-09T09:35:29.709 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2040", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:31:54 UTC", "S 2021-03-09T09:35:29.710 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "23560162", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:31:54 UTC", "WatchdogTimestampMonotonic": "23560161", "WatchdogUSec": "0"}} 2021-03-09T09:35:29.738 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.051) 0:02:24.422 ********* 2021-03-09T09:35:29.766 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:29.768 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-09T09:35:29.790 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.051) 0:02:24.474 ********* 2021-03-09T09:35:29.818 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:29.819 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:29.841 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.050) 0:02:24.525 ********* 2021-03-09T09:35:29.868 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-09T09:35:29.870 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:29.892 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:29 +0000 (0:00:00.050) 0:02:24.576 ********* 2021-03-09T09:35:30.556 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-09T09:35:30.558 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:35:30.581 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:30 +0000 (0:00:00.689) 0:02:25.265 ********* 2021-03-09T09:35:30.608 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-09T09:35:30.610 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:30.632 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:30 +0000 (0:00:00.051) 0:02:25.316 ********* 2021-03-09T09:35:30.774 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-09T09:35:30.776 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T09:35:30.798 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:30 +0000 (0:00:00.166) 0:02:25.483 ********* 2021-03-09T09:35:30.836 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-09T09:35:30.838 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} 2021-03-09T09:35:30.864 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:30 +0000 (0:00:00.065) 0:02:25.548 ********* 2021-03-09T09:35:30.892 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-09T09:35:30.894 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:30.916 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:30 +0000 (0:00:00.051) 0:02:25.600 ********* 2021-03-09T09:35:31.068 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-09T09:35:31.069 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T09:35:31.092 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:31 +0000 (0:00:00.175) 0:02:25.776 ********* 2021-03-09T09:35:31.129 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-09T09:35:31.131 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-09T09:35:31.160 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:31 +0000 (0:00:00.068) 0:02:25.844 ********* 2021-03-09T09:35:31.431 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-09T09:35:31.456 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:31 +0000 (0:00:00.295) 0:02:26.140 ********* 2021-03-09T09:35:32.228 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-09T09:35:32.229 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-09T09:35:32.251 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:32 +0000 (0:00:00.795) 0:02:26.935 ********* 2021-03-09T09:35:32.259 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-09T09:35:32.295 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:32 +0000 (0:00:00.043) 0:02:26.979 ********* 2021-03-09T09:35:32.770 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-09T09:35:32.771 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false} 2021-03-09T09:35:32.796 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:32 +0000 (0:00:00.500) 0:02:27.480 ********* 2021-03-09T09:35:32.823 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-09T09:35:32.825 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-09T09:35:32.847 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:32 +0000 (0:00:00.050) 0:02:27.531 ********* 2021-03-09T09:35:32.864 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-09T09:35:32.865 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false} 2021-03-09T09:35:32.894 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:32 +0000 (0:00:00.047) 0:02:27.578 ********* 2021-03-09T09:35:32.923 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-09T09:35:32.925 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-03-09T09:35:32.952 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:32 +0000 (0:00:00.057) 0:02:27.636 ********* 2021-03-09T09:35:32.991 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-09T09:35:32.992 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1615282356.6876526, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1615282356.6836526, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1615282356.6836526, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-09T09:35:33.024 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.072) 0:02:27.708 ********* 2021-03-09T09:35:33.115 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1615282323.369608, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1615282323.365608, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1615282323.365608, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-09T09:35:33.141 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.117) 0:02:27.825 ********* 2021-03-09T09:35:33.441 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-09T09:35:33.443 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-09T09:35:33.469 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.327) 0:02:28.153 ********* 2021-03-09T09:35:33.476 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-09T09:35:33.502 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.033) 0:02:28.186 ********* 2021-03-09T09:35:33.509 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-09T09:35:33.510 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-09T09:35:33.535 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.033) 0:02:28.219 ********* 2021-03-09T09:35:33.541 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-09T09:35:33.566 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.031) 0:02:28.250 ********* 2021-03-09T09:35:33.573 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-09T09:35:33.575 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:33.600 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.033) 0:02:28.284 ********* 2021-03-09T09:35:33.607 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:33.634 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.033) 0:02:28.318 ********* 2021-03-09T09:35:33.735 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-09T09:35:33.737 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-09T09:35:33.760 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.126) 0:02:28.444 ********* 2021-03-09T09:35:33.777 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-09T09:35:33.803 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.043) 0:02:28.487 ********* 2021-03-09T09:35:33.894 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-09T09:35:33.896 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-09T09:35:33.922 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.118) 0:02:28.606 ********* 2021-03-09T09:35:33.941 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-09T09:35:33.974 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:33 +0000 (0:00:00.051) 0:02:28.658 ********* 2021-03-09T09:35:33.981 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-09T09:35:33.983 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.006 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.032) 0:02:28.690 ********* 2021-03-09T09:35:34.012 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.039 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.033) 0:02:28.723 ********* 2021-03-09T09:35:34.048 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-09T09:35:34.050 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.060 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.062 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi194.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.088 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.048) 0:02:28.772 ********* 2021-03-09T09:35:34.095 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.097 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.124 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.035) 0:02:28.808 ********* 2021-03-09T09:35:34.132 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-09T09:35:34.134 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.135 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.145 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.157 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.159 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.169 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.170 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.172 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.195 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.071) 0:02:28.879 ********* 2021-03-09T09:35:34.203 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.204 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.235 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.039) 0:02:28.919 ********* 2021-03-09T09:35:34.243 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:35:34.245 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.271 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.035) 0:02:28.955 ********* 2021-03-09T09:35:34.278 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.305 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.033) 0:02:28.989 ********* 2021-03-09T09:35:34.312 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-09T09:35:34.313 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.339 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.034) 0:02:29.023 ********* 2021-03-09T09:35:34.349 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.385 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.045) 0:02:29.069 ********* 2021-03-09T09:35:34.393 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-09T09:35:34.395 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.420 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.034) 0:02:29.104 ********* 2021-03-09T09:35:34.427 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.457 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.037) 0:02:29.141 ********* 2021-03-09T09:35:34.464 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-09T09:35:34.466 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.489 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.031) 0:02:29.173 ********* 2021-03-09T09:35:34.495 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.521 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.032) 0:02:29.205 ********* 2021-03-09T09:35:34.527 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-09T09:35:34.529 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.555 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.033) 0:02:29.239 ********* 2021-03-09T09:35:34.564 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.591 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.036) 0:02:29.275 ********* 2021-03-09T09:35:34.598 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T09:35:34.599 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.625 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.033) 0:02:29.309 ********* 2021-03-09T09:35:34.631 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.666 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.041) 0:02:29.350 ********* 2021-03-09T09:35:34.673 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-09T09:35:34.675 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.701 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.034) 0:02:29.385 ********* 2021-03-09T09:35:34.708 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.735 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.033) 0:02:29.419 ********* 2021-03-09T09:35:34.741 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T09:35:34.743 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.767 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.031) 0:02:29.451 ********* 2021-03-09T09:35:34.774 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:34.801 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:34 +0000 (0:00:00.034) 0:02:29.485 ********* 2021-03-09T09:35:34.924 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-09T09:35:34.926 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011070", "end": "2021-03-09 09:35:34.863683", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-09 09:35:34.852613", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:34.967 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011026", "end": "2021-03-09 09:35:34.934249", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-09 09:35:34.923223", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:35.051 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006079", "end": "2021-03-09 09:35:34.986513", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-09 09:35:34.980434", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:35.083 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006089", "end": "2021-03-09 09:35:35.060593", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-09 09:35:35.054504", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:35.166 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005437", "end": "2021-03-09 09:35:35.108282", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 09:35:35.102845", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:35.219 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005550", "end": "2021-03-09 09:35:35.183379", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 09:35:35.177829", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:35.292 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.015997", "end": "2021-03-09 09:35:35.234315", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-09 09:35:35.218318", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:35.333 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:35 +0000 (0:00:00.532) 0:02:30.017 ********* 2021-03-09T09:35:35.351 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.018186", "end": "2021-03-09 09:35:35.323768", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-09 09:35:35.305582", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:35.377 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:35 +0000 (0:00:00.043) 0:02:30.061 ********* 2021-03-09T09:35:35.384 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-09T09:35:35.409 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:35 +0000 (0:00:00.032) 0:02:30.093 ********* 2021-03-09T09:35:35.444 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:35 +0000 (0:00:00.034) 0:02:30.128 ********* 2021-03-09T09:35:35.545 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-09T09:35:35.547 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-09T09:35:35.595 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-09T09:35:35.666 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-03-09T09:35:35.685 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:35 +0000 (0:00:00.241) 0:02:30.369 ********* 2021-03-09T09:35:35.706 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:35 +0000 (0:00:00.020) 0:02:30.390 ********* 2021-03-09T09:35:36.442 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi194.front.sepia.ceph.com] 2021-03-09T09:35:36.475 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] 2021-03-09T09:35:36.514 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:36 +0000 (0:00:00.807) 0:02:31.198 ********* 2021-03-09T09:35:36.547 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:36 +0000 (0:00:00.032) 0:02:31.231 ********* 2021-03-09T09:35:36.553 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:35:36.555 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-09T09:35:36.579 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:36 +0000 (0:00:00.032) 0:02:31.263 ********* 2021-03-09T09:35:36.587 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-09T09:35:36.612 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:36 +0000 (0:00:00.032) 0:02:31.296 ********* 2021-03-09T09:35:36.619 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:35:36.620 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:36.644 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:36 +0000 (0:00:00.032) 0:02:31.328 ********* 2021-03-09T09:35:36.651 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:36.680 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:36 +0000 (0:00:00.035) 0:02:31.364 ********* 2021-03-09T09:35:36.699 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-09T09:35:36.701 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-09T09:35:36.733 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:36 +0000 (0:00:00.053) 0:02:31.417 ********* 2021-03-09T09:35:36.740 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-09T09:35:36.767 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:36 +0000 (0:00:00.034) 0:02:31.451 ********* 2021-03-09T09:35:37.609 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-09T09:35:37.610 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"cache_update_time": 1615282376, "cache_updated": false, "changed": false} 2021-03-09T09:35:37.633 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:37 +0000 (0:00:00.865) 0:02:32.317 ********* 2021-03-09T09:35:37.659 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:35:37.661 INFO:teuthology.task.ansible.out:skipping: [smithi194.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:37.685 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:37 +0000 (0:00:00.052) 0:02:32.369 ********* 2021-03-09T09:35:37.714 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-09T09:35:37.716 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1615282371, "cache_updated": false, "changed": false} TASK [container-host : set_fact] *********************************************** ok: [smithi194.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-09T09:35:37.739 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:37 +0000 (0:00:00.054) 0:02:32.423 ********* 2021-03-09T09:35:37.764 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:37 +0000 (0:00:00.024) 0:02:32.448 ********* 2021-03-09T09:35:37.769 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:35:37.771 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:35:37.798 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:37 +0000 (0:00:00.033) 0:02:32.482 ********* 2021-03-09T09:35:37.841 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:35:37.842 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-09T09:35:37.865 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:37 +0000 (0:00:00.066) 0:02:32.549 ********* 2021-03-09T09:35:37.913 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-09T09:35:37.915 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:35:38.024 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:35:38.027 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:35:38.053 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:38 +0000 (0:00:00.188) 0:02:32.737 ********* 2021-03-09T09:35:38.154 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:35:38.180 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:38 +0000 (0:00:00.127) 0:02:32.864 ********* 2021-03-09T09:35:38.396 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-09T09:35:38.398 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282538.111855-13198-95267289025138/source", "state": "file", "uid": 0} 2021-03-09T09:35:38.543 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282538.2386627-13207-220156368338719/source", "state": "file", "uid": 0} 2021-03-09T09:35:38.710 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282538.4210718-13198-122452284040094/source", "state": "file", "uid": 0} 2021-03-09T09:35:38.736 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:38 +0000 (0:00:00.555) 0:02:33.420 ********* 2021-03-09T09:35:38.878 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282538.572276-13207-58424771506978/source", "state": "file", "uid": 0} 2021-03-09T09:35:38.904 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:38 +0000 (0:00:00.167) 0:02:33.588 ********* 2021-03-09T09:35:42.276 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-09T09:35:42.278 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-NYTdI9-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-NYTdI9-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-09T09:35:42.279 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-09T09:35:42.310 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:42 +0000 (0:00:03.406) 0:02:36.995 ********* 2021-03-09T09:35:42.466 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-09T09:35:42.468 INFO:teuthology.task.ansible.out:ok: [smithi194.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-09T09:35:42.494 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:42 +0000 (0:00:00.183) 0:02:37.178 ********* 2021-03-09T09:35:42.575 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-09T09:35:42.577 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-EWhrbo-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-EWhrbo-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-09T09:35:42.579 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-09T09:35:42.604 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:42 +0000 (0:00:00.109) 0:02:37.288 ********* 2021-03-09T09:35:42.746 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-09T09:35:42.748 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-09T09:35:42.770 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:42 +0000 (0:00:00.166) 0:02:37.455 ********* 2021-03-09T09:35:42.872 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-03-09T09:35:42.874 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282542.5533442-13452-33573653035375/source", "state": "file", "uid": 0} 2021-03-09T09:35:42.898 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:42 +0000 (0:00:00.127) 0:02:37.582 ********* 2021-03-09T09:35:43.114 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282542.828412-13487-217827092886213/source", "state": "file", "uid": 0} 2021-03-09T09:35:43.143 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:43 +0000 (0:00:00.245) 0:02:37.827 ********* 2021-03-09T09:35:43.161 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-09T09:35:43.163 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.119351", "end": "2021-03-09 09:35:43.104929", "rc": 0, "start": "2021-03-09 09:35:42.985578", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:43.187 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:43 +0000 (0:00:00.043) 0:02:37.871 ********* 2021-03-09T09:35:43.402 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.119953", "end": "2021-03-09 09:35:43.379247", "rc": 0, "start": "2021-03-09 09:35:43.259294", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:35:43.426 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:35:43 +0000 (0:00:00.239) 0:02:38.110 ********* 2021-03-09T09:35:46.364 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-09T09:35:46.365 INFO:teuthology.task.ansible.out:changed: [smithi194.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target containerd.service firewalld.service system.slice systemd-journald.socket sysinit.target network-online.target docker.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-09T09:35:46.366 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "12802 2021-03-09T09:35:46.368 INFO:teuthology.task.ansible.out:9", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket system.slice sysinit.target", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:32:39 UTC", "StateChangeTimestampMonotonic": "40700255", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "Su 2021-03-09T09:35:46.370 INFO:teuthology.task.ansible.out:ccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-online.target containerd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T09:35:46.646 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "network-online.target sysinit.target system.slice basic.target docker.socket containerd.service firewalld.service systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-09T09:35:46.648 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "12805 2021-03-09T09:35:46.649 INFO:teuthology.task.ansible.out:3", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice docker.socket", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:32:11 UTC", "StateChangeTimestampMonotonic": "40957201", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "Su 2021-03-09T09:35:46.651 INFO:teuthology.task.ansible.out:ccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-online.target containerd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T09:35:46.716 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-03-09T09:35:46.717 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-03-09T09:35:46.719 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi052.front.sepia.ceph.com : ok=127 changed=35 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi194.front.sepia.ceph.com : ok=127 changed=35 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Tuesday 09 March 2021 09:35:46 +0000 (0:00:03.292) 0:02:41.403 ********* =============================================================================== 2021-03-09T09:35:46.719 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 20.48s users : Update authorized_keys using the keys repo --------------------- 18.18s testnode : Ensure packages are not present. ---------------------------- 10.28s testnode : Install packages --------------------------------------------- 8.41s testnode : Zap all non-root disks --------------------------------------- 8.19s Remove revoked users ---------------------------------------------------- 6.82s container-host : Install registries-conf-ctl ---------------------------- 3.41s common : Update apt cache ----------------------------------------------- 3.30s 2021-03-09T09:35:46.720 INFO:teuthology.task.ansible.out:container-host : Restart docker service --------------------------------- 3.29s testnode : Update apt cache. -------------------------------------------- 2.96s testnode : Install apt keys --------------------------------------------- 2.32s Gathering Facts --------------------------------------------------------- 1.83s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.66s testnode : Create logical volume(s) ------------------------------------- 1.36s users : Clone the keys repo --------------------------------------------- 1.35s users : Update authorized_keys for each user with literal keys ---------- 1.35s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.00s tes 2021-03-09T09:35:46.721 INFO:teuthology.task.ansible.out:tnode : Set the system's hostname ------------------------------------ 0.94s common : Make sure smartmontools is installed --------------------------- 0.94s testnode : Install python-apt ------------------------------------------- 0.92s 2021-03-09T09:35:47.076 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2021-03-09T09:35:47.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T09:35:47.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T09:35:47.160 DEBUG:teuthology.orchestra.run.smithi052:> true 2021-03-09T09:35:47.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T09:35:47.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T09:35:47.439 DEBUG:teuthology.orchestra.run.smithi194:> true 2021-03-09T09:35:47.631 INFO:teuthology.run_tasks:Running task clock... 2021-03-09T09:35:47.645 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-03-09T09:35:47.646 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-09T09:35:47.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-09T09:35:47.707 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-09T09:35:47.708 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: Command line: ntpd -gq 2021-03-09T09:35:47.708 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: proto: precision = 0.127 usec (-23) 2021-03-09T09:35:47.709 INFO:teuthology.orchestra.run.smithi052.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T09:35:47.709 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T09:35:47.710 INFO:teuthology.orchestra.run.smithi052.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-09T09:35:47.711 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: restrict ::: KOD does nothing without LIMITED. 2021-03-09T09:35:47.711 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: Listen and drop on 0 v6wildcard [::]:123 2021-03-09T09:35:47.711 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-09T09:35:47.712 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: Listen normally on 2 lo 127.0.0.1:123 2021-03-09T09:35:47.712 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: Listen normally on 3 ens1f0 172.21.15.52:123 2021-03-09T09:35:47.712 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: Listen normally on 4 lo [::1]:123 2021-03-09T09:35:47.712 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15de%3]:123 2021-03-09T09:35:47.713 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:47 ntpd[9560]: Listening on routing socket on fd #22 for interface updates 2021-03-09T09:35:48.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:48 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:48.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:48 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:48.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:48 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:48.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:48 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:49.708 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:49 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:49.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:49 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:49.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:49 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:49.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:49 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:50.708 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:50 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:50.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:50 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:50.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:50 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:50.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:50 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:50.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:50 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:50.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:50 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:50.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:50 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:50.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:50 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:51.708 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:51 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:51.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:51 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:51.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:51 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:51.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:51 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:51.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:51 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:51.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:51 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:51.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:51 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:51.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:51 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:52.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:52 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:52.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:52 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:52.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:52 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:52.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:52 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:52.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:52 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:52.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:52 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:52.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:52 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:52.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:52 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:53.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:53 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:53.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:53 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:53.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:53 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:53.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:53 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:53.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:53 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:53.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:53 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:53.711 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:53 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:53.711 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:53 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:54.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:54 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:54.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:54 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:54.709 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:54 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:54.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:54 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:54.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:54 ntpd[9560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:54.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:54 ntpd[9560]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:54.710 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:54 ntpd[9560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:54.711 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:54 ntpd[9560]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:54.711 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Mar 09:35:54 ntpd[9560]: ntpd: time slew +0.000429 s 2021-03-09T09:35:54.711 INFO:teuthology.orchestra.run.smithi052.stdout:ntpd: time slew +0.000429s 2021-03-09T09:35:54.712 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:54 ntpd[9560]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-09T09:35:54.712 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Mar 09:35:54 ntpd[9560]: can't open /var/log/ntpstats/loopstats.20210309: Permission denied 2021-03-09T09:35:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: remote refid st t when poll reach delay offset jitter 2021-03-09T09:35:54.777 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================== 2021-03-09T09:35:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:35:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:35:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:35:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:35:54.779 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-09T09:35:54.780 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-09T09:35:54.832 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-09T09:35:54.832 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: Command line: ntpd -gq 2021-03-09T09:35:54.833 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: proto: precision = 0.123 usec (-23) 2021-03-09T09:35:54.833 INFO:teuthology.orchestra.run.smithi194.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T09:35:54.834 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T09:35:54.835 INFO:teuthology.orchestra.run.smithi194.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-09T09:35:54.836 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: restrict ::: KOD does nothing without LIMITED. 2021-03-09T09:35:54.836 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: Listen and drop on 0 v6wildcard [::]:123 2021-03-09T09:35:54.836 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-09T09:35:54.836 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: Listen normally on 2 lo 127.0.0.1:123 2021-03-09T09:35:54.837 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: Listen normally on 3 enp3s0f1 172.21.15.194:123 2021-03-09T09:35:54.837 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: Listen normally on 4 lo [::1]:123 2021-03-09T09:35:54.837 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ea5%5]:123 2021-03-09T09:35:54.837 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:35:54 ntpd[9215]: Listening on routing socket on fd #22 for interface updates 2021-03-09T09:35:55.832 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:55 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:55.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:55 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:55.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:55 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:55.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:55 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:56.832 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:56 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:56.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:56 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:56.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:56 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:56.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:56 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:56.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:56 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:56.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:56 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:56.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:56 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:56.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:56 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:57.832 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:57 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:57.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:57 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:57.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:57 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:57.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:57 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:58.832 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:58 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:58.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:58 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:58.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:58 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:58.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:58 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:58.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:58 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:58.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:58 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:58.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:58 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:58.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:58 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:59.832 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:59 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:59.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:59 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:59.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:59 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:59.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:59 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:35:59.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:59 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:35:59.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:59 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:35:59.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:59 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:35:59.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:35:59 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:36:00.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:00 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:36:00.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:00 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:36:00.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:00 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:36:00.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:00 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:36:00.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:00 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:36:00.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:00 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:36:00.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:00 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:36:00.835 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:00 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:36:01.832 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:01 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:36:01.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:01 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:36:01.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:01 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:36:01.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:01 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:36:01.833 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:01 ntpd[9215]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:36:01.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:01 ntpd[9215]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:36:01.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:01 ntpd[9215]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:36:01.834 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:01 ntpd[9215]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:36:01.835 INFO:teuthology.orchestra.run.smithi194.stdout: 9 Mar 09:36:01 ntpd[9215]: ntpd: time set +0.033190 s 2021-03-09T09:36:01.835 INFO:teuthology.orchestra.run.smithi194.stdout:ntpd: time set +0.033190s 2021-03-09T09:36:01.835 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:01 ntpd[9215]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-09T09:36:01.836 INFO:teuthology.orchestra.run.smithi194.stderr: 9 Mar 09:36:01 ntpd[9215]: can't open /var/log/ntpstats/loopstats.20210309: Permission denied 2021-03-09T09:36:01.946 INFO:teuthology.orchestra.run.smithi194.stdout: remote refid st t when poll reach delay offset jitter 2021-03-09T09:36:01.947 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================== 2021-03-09T09:36:01.947 INFO:teuthology.orchestra.run.smithi194.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:36:01.947 INFO:teuthology.orchestra.run.smithi194.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:36:01.947 INFO:teuthology.orchestra.run.smithi194.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:36:01.948 INFO:teuthology.orchestra.run.smithi194.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:36:01.948 INFO:teuthology.run_tasks:Running task install... 2021-03-09T09:36:01.959 DEBUG:teuthology.task.install:project ceph 2021-03-09T09:36:01.963 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}} 2021-03-09T09:36:01.963 DEBUG:teuthology.task.install:config {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'} 2021-03-09T09:36:01.963 INFO:teuthology.task.install:Using flavor: basic 2021-03-09T09:36:01.973 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-09T09:36:01.974 INFO:teuthology.task.install:extra packages: [] 2021-03-09T09:36:01.975 DEBUG:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2021-03-09T09:36:02.028 INFO:teuthology.orchestra.run.smithi052.stdout:Ubuntu 2021-03-09T09:36:02.033 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:36:02.034 DEBUG:teuthology.orchestra.run.smithi194:> sudo lsb_release -is 2021-03-09T09:36:02.035 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-key list | grep Ceph 2021-03-09T09:36:02.092 INFO:teuthology.orchestra.run.smithi194.stdout:Ubuntu 2021-03-09T09:36:02.098 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:36:02.099 DEBUG:teuthology.orchestra.run.smithi194:> sudo apt-key list | grep Ceph 2021-03-09T09:36:02.135 INFO:teuthology.orchestra.run.smithi052.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-09T09:36:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-09T09:36:02.192 INFO:teuthology.orchestra.run.smithi052.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-09T09:36:02.192 INFO:teuthology.orchestra.run.smithi052.stdout:uid [ unknown] Ceph.com (release key) 2021-03-09T09:36:02.194 DEBUG:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2021-03-09T09:36:02.246 INFO:teuthology.orchestra.run.smithi194.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-09T09:36:02.247 INFO:teuthology.orchestra.run.smithi194.stdout:uid [ unknown] Ceph.com (release key) 2021-03-09T09:36:02.248 DEBUG:teuthology.orchestra.run.smithi194:> sudo lsb_release -is 2021-03-09T09:36:02.249 INFO:teuthology.orchestra.run.smithi052.stdout:Ubuntu 2021-03-09T09:36:02.254 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:36:02.254 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-09T09:36:02.255 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T09:36:02.299 INFO:teuthology.orchestra.run.smithi194.stdout:Ubuntu 2021-03-09T09:36:02.304 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:36:02.304 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-09T09:36:02.304 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T09:36:02.414 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default/ 2021-03-09T09:36:02.414 INFO:teuthology.task.install.deb:Package version is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:36:02.456 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default/ 2021-03-09T09:36:02.457 INFO:teuthology.task.install.deb:Package version is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:36:02.533 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:36:02.533 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-09T09:36:02.549 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get update 2021-03-09T09:36:02.573 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:36:02.573 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-09T09:36:02.588 DEBUG:teuthology.orchestra.run.smithi194:> sudo apt-get update 2021-03-09T09:36:02.737 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-09T09:36:02.772 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-09T09:36:02.799 INFO:teuthology.orchestra.run.smithi052.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic InRelease 2021-03-09T09:36:02.822 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-03-09T09:36:02.837 INFO:teuthology.orchestra.run.smithi194.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic InRelease 2021-03-09T09:36:02.847 INFO:teuthology.orchestra.run.smithi052.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-09T09:36:02.851 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T09:36:02.861 INFO:teuthology.orchestra.run.smithi194.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-03-09T09:36:02.870 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,099 B] 2021-03-09T09:36:02.884 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T09:36:02.887 INFO:teuthology.orchestra.run.smithi194.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-09T09:36:02.905 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-03-09T09:36:02.911 INFO:teuthology.orchestra.run.smithi194.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,099 B] 2021-03-09T09:36:02.931 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-09T09:36:02.946 INFO:teuthology.orchestra.run.smithi194.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-03-09T09:36:02.962 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-09T09:36:03.011 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-09T09:36:03.041 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-09T09:36:03.347 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 27.4 kB in 1s (46.3 kB/s) 2021-03-09T09:36:03.369 INFO:teuthology.orchestra.run.smithi194.stdout:Fetched 27.4 kB in 1s (47.4 kB/s) 2021-03-09T09:36:04.615 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2021-03-09T09:36:04.644 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-09T09:36:04.660 DEBUG:teuthology.orchestra.run.smithi194:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-1742-g89fb622b-1bionic cephadm=17.0.0-1742-g89fb622b-1bionic ceph-mds=17.0.0-1742-g89fb622b-1bionic ceph-mgr=17.0.0-1742-g89fb622b-1bionic ceph-common=17.0.0-1742-g89fb622b-1bionic ceph-fuse=17.0.0-1742-g89fb622b-1bionic ceph-test=17.0.0-1742-g89fb622b-1bionic radosgw=17.0.0-1742-g89fb622b-1bionic python3-rados=17.0.0-1742-g89fb622b-1bionic python3-rgw=17.0.0-1742-g89fb622b-1bionic python3-cephfs=17.0.0-1742-g89fb622b-1bionic python3-rbd=17.0.0-1742-g89fb622b-1bionic libcephfs2=17.0.0-1742-g89fb622b-1bionic libcephfs-dev=17.0.0-1742-g89fb622b-1bionic librados2=17.0.0-1742-g89fb622b-1bionic librbd1=17.0.0-1742-g89fb622b-1bionic rbd-fuse=17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:36:04.692 DEBUG:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-1742-g89fb622b-1bionic cephadm=17.0.0-1742-g89fb622b-1bionic ceph-mds=17.0.0-1742-g89fb622b-1bionic ceph-mgr=17.0.0-1742-g89fb622b-1bionic ceph-common=17.0.0-1742-g89fb622b-1bionic ceph-fuse=17.0.0-1742-g89fb622b-1bionic ceph-test=17.0.0-1742-g89fb622b-1bionic radosgw=17.0.0-1742-g89fb622b-1bionic python3-rados=17.0.0-1742-g89fb622b-1bionic python3-rgw=17.0.0-1742-g89fb622b-1bionic python3-cephfs=17.0.0-1742-g89fb622b-1bionic python3-rbd=17.0.0-1742-g89fb622b-1bionic libcephfs2=17.0.0-1742-g89fb622b-1bionic libcephfs-dev=17.0.0-1742-g89fb622b-1bionic librados2=17.0.0-1742-g89fb622b-1bionic librbd1=17.0.0-1742-g89fb622b-1bionic rbd-fuse=17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:36:04.742 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2021-03-09T09:36:04.777 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-09T09:36:05.003 INFO:teuthology.orchestra.run.smithi194.stdout:Building dependency tree... 2021-03-09T09:36:05.004 INFO:teuthology.orchestra.run.smithi194.stdout:Reading state information... 2021-03-09T09:36:05.051 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2021-03-09T09:36:05.052 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2021-03-09T09:36:05.171 INFO:teuthology.orchestra.run.smithi194.stdout:The following additional packages will be installed: 2021-03-09T09:36:05.171 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-03-09T09:36:05.173 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-03-09T09:36:05.173 INFO:teuthology.orchestra.run.smithi194.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-09T09:36:05.174 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-09T09:36:05.174 INFO:teuthology.orchestra.run.smithi194.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-09T09:36:05.174 INFO:teuthology.orchestra.run.smithi194.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T09:36:05.174 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-09T09:36:05.175 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-09T09:36:05.175 INFO:teuthology.orchestra.run.smithi194.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-09T09:36:05.175 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-09T09:36:05.175 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-09T09:36:05.175 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-09T09:36:05.176 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-09T09:36:05.176 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-09T09:36:05.176 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-09T09:36:05.176 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-09T09:36:05.177 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-09T09:36:05.177 INFO:teuthology.orchestra.run.smithi194.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-09T09:36:05.177 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-09T09:36:05.177 INFO:teuthology.orchestra.run.smithi194.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-09T09:36:05.177 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-09T09:36:05.178 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-09T09:36:05.179 INFO:teuthology.orchestra.run.smithi194.stdout:Suggested packages: 2021-03-09T09:36:05.179 INFO:teuthology.orchestra.run.smithi194.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-09T09:36:05.179 INFO:teuthology.orchestra.run.smithi194.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-09T09:36:05.179 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-09T09:36:05.179 INFO:teuthology.orchestra.run.smithi194.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-09T09:36:05.180 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-09T09:36:05.180 INFO:teuthology.orchestra.run.smithi194.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-09T09:36:05.180 INFO:teuthology.orchestra.run.smithi194.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-09T09:36:05.180 INFO:teuthology.orchestra.run.smithi194.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-09T09:36:05.181 INFO:teuthology.orchestra.run.smithi194.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-09T09:36:05.181 INFO:teuthology.orchestra.run.smithi194.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-09T09:36:05.181 INFO:teuthology.orchestra.run.smithi194.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-09T09:36:05.181 INFO:teuthology.orchestra.run.smithi194.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-09T09:36:05.181 INFO:teuthology.orchestra.run.smithi194.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-09T09:36:05.182 INFO:teuthology.orchestra.run.smithi194.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-09T09:36:05.182 INFO:teuthology.orchestra.run.smithi194.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-09T09:36:05.235 INFO:teuthology.orchestra.run.smithi052.stdout:The following additional packages will be installed: 2021-03-09T09:36:05.236 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-03-09T09:36:05.237 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-03-09T09:36:05.237 INFO:teuthology.orchestra.run.smithi052.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-09T09:36:05.237 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-09T09:36:05.238 INFO:teuthology.orchestra.run.smithi052.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-09T09:36:05.238 INFO:teuthology.orchestra.run.smithi052.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T09:36:05.238 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-09T09:36:05.238 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-09T09:36:05.239 INFO:teuthology.orchestra.run.smithi052.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-09T09:36:05.239 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-09T09:36:05.239 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-09T09:36:05.239 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-09T09:36:05.239 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-09T09:36:05.240 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-09T09:36:05.240 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-09T09:36:05.240 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-09T09:36:05.240 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-09T09:36:05.240 INFO:teuthology.orchestra.run.smithi052.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-09T09:36:05.241 INFO:teuthology.orchestra.run.smithi052.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-09T09:36:05.241 INFO:teuthology.orchestra.run.smithi052.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-09T09:36:05.241 INFO:teuthology.orchestra.run.smithi052.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-09T09:36:05.241 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-09T09:36:05.242 INFO:teuthology.orchestra.run.smithi052.stdout:Suggested packages: 2021-03-09T09:36:05.242 INFO:teuthology.orchestra.run.smithi052.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-09T09:36:05.242 INFO:teuthology.orchestra.run.smithi052.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-09T09:36:05.242 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-09T09:36:05.243 INFO:teuthology.orchestra.run.smithi052.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-09T09:36:05.243 INFO:teuthology.orchestra.run.smithi052.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-09T09:36:05.243 INFO:teuthology.orchestra.run.smithi052.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-09T09:36:05.243 INFO:teuthology.orchestra.run.smithi052.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-09T09:36:05.243 INFO:teuthology.orchestra.run.smithi052.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-09T09:36:05.244 INFO:teuthology.orchestra.run.smithi052.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-09T09:36:05.244 INFO:teuthology.orchestra.run.smithi052.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-09T09:36:05.244 INFO:teuthology.orchestra.run.smithi052.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-09T09:36:05.244 INFO:teuthology.orchestra.run.smithi052.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-09T09:36:05.244 INFO:teuthology.orchestra.run.smithi052.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-09T09:36:05.245 INFO:teuthology.orchestra.run.smithi052.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-09T09:36:05.245 INFO:teuthology.orchestra.run.smithi052.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-09T09:36:05.270 INFO:teuthology.orchestra.run.smithi194.stdout:The following NEW packages will be installed: 2021-03-09T09:36:05.270 INFO:teuthology.orchestra.run.smithi194.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-09T09:36:05.271 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-03-09T09:36:05.271 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-03-09T09:36:05.273 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-03-09T09:36:05.273 INFO:teuthology.orchestra.run.smithi194.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-09T09:36:05.274 INFO:teuthology.orchestra.run.smithi194.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-09T09:36:05.274 INFO:teuthology.orchestra.run.smithi194.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-09T09:36:05.274 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T09:36:05.275 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-09T09:36:05.275 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-09T09:36:05.275 INFO:teuthology.orchestra.run.smithi194.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-09T09:36:05.275 INFO:teuthology.orchestra.run.smithi194.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-09T09:36:05.275 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-09T09:36:05.276 INFO:teuthology.orchestra.run.smithi194.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-09T09:36:05.276 INFO:teuthology.orchestra.run.smithi194.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-09T09:36:05.276 INFO:teuthology.orchestra.run.smithi194.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-09T09:36:05.276 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-09T09:36:05.276 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-09T09:36:05.277 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-09T09:36:05.277 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-09T09:36:05.277 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-09T09:36:05.277 INFO:teuthology.orchestra.run.smithi194.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-09T09:36:05.277 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-09T09:36:05.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-09T09:36:05.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-09T09:36:05.278 INFO:teuthology.orchestra.run.smithi194.stdout:The following packages will be upgraded: 2021-03-09T09:36:05.279 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 librbd1 2021-03-09T09:36:05.346 INFO:teuthology.orchestra.run.smithi052.stdout:The following NEW packages will be installed: 2021-03-09T09:36:05.346 INFO:teuthology.orchestra.run.smithi052.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-09T09:36:05.346 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-03-09T09:36:05.348 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-03-09T09:36:05.348 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-03-09T09:36:05.348 INFO:teuthology.orchestra.run.smithi052.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-09T09:36:05.348 INFO:teuthology.orchestra.run.smithi052.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-09T09:36:05.349 INFO:teuthology.orchestra.run.smithi052.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-09T09:36:05.349 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T09:36:05.349 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-09T09:36:05.349 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-09T09:36:05.350 INFO:teuthology.orchestra.run.smithi052.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-09T09:36:05.350 INFO:teuthology.orchestra.run.smithi052.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-09T09:36:05.350 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-09T09:36:05.350 INFO:teuthology.orchestra.run.smithi052.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-09T09:36:05.350 INFO:teuthology.orchestra.run.smithi052.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-09T09:36:05.351 INFO:teuthology.orchestra.run.smithi052.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-09T09:36:05.351 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-09T09:36:05.351 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-09T09:36:05.351 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-09T09:36:05.351 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-09T09:36:05.352 INFO:teuthology.orchestra.run.smithi052.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-09T09:36:05.352 INFO:teuthology.orchestra.run.smithi052.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-09T09:36:05.352 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-09T09:36:05.352 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-09T09:36:05.352 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-09T09:36:05.353 INFO:teuthology.orchestra.run.smithi052.stdout:The following packages will be upgraded: 2021-03-09T09:36:05.354 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 librbd1 2021-03-09T09:36:05.410 INFO:teuthology.orchestra.run.smithi194.stdout:2 upgraded, 111 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T09:36:05.411 INFO:teuthology.orchestra.run.smithi194.stdout:Need to get 166 MB of archives. 2021-03-09T09:36:05.411 INFO:teuthology.orchestra.run.smithi194.stdout:After this operation, 778 MB of additional disk space will be used. 2021-03-09T09:36:05.411 INFO:teuthology.orchestra.run.smithi194.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-1742-g89fb622b-1bionic [3,728 B] 2021-03-09T09:36:05.456 INFO:teuthology.orchestra.run.smithi194.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-1742-g89fb622b-1bionic [3,126 kB] 2021-03-09T09:36:05.459 INFO:teuthology.orchestra.run.smithi194.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-09T09:36:05.488 INFO:teuthology.orchestra.run.smithi052.stdout:2 upgraded, 111 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T09:36:05.488 INFO:teuthology.orchestra.run.smithi052.stdout:Need to get 166 MB of archives. 2021-03-09T09:36:05.489 INFO:teuthology.orchestra.run.smithi052.stdout:After this operation, 778 MB of additional disk space will be used. 2021-03-09T09:36:05.489 INFO:teuthology.orchestra.run.smithi052.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-1742-g89fb622b-1bionic [3,728 B] 2021-03-09T09:36:05.534 INFO:teuthology.orchestra.run.smithi052.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-1742-g89fb622b-1bionic [3,126 kB] 2021-03-09T09:36:05.538 INFO:teuthology.orchestra.run.smithi194.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-09T09:36:05.540 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-09T09:36:05.620 INFO:teuthology.orchestra.run.smithi052.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-09T09:36:05.788 INFO:teuthology.orchestra.run.smithi194.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-09T09:36:05.797 INFO:teuthology.orchestra.run.smithi194.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-09T09:36:05.814 INFO:teuthology.orchestra.run.smithi194.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-09T09:36:05.841 INFO:teuthology.orchestra.run.smithi194.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-1742-g89fb622b-1bionic [3,169 kB] 2021-03-09T09:36:05.877 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-09T09:36:05.884 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-09T09:36:05.898 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-09T09:36:05.910 INFO:teuthology.orchestra.run.smithi194.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-09T09:36:05.913 INFO:teuthology.orchestra.run.smithi194.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-09T09:36:05.942 INFO:teuthology.orchestra.run.smithi194.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-09T09:36:05.950 INFO:teuthology.orchestra.run.smithi194.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-09T09:36:05.966 INFO:teuthology.orchestra.run.smithi194.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-09T09:36:05.975 INFO:teuthology.orchestra.run.smithi194.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-09T09:36:05.981 INFO:teuthology.orchestra.run.smithi194.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-09T09:36:05.999 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-09T09:36:06.001 INFO:teuthology.orchestra.run.smithi194.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-09T09:36:06.003 INFO:teuthology.orchestra.run.smithi052.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-09T09:36:06.006 INFO:teuthology.orchestra.run.smithi194.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-09T09:36:06.025 INFO:teuthology.orchestra.run.smithi194.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-09T09:36:06.033 INFO:teuthology.orchestra.run.smithi052.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-09T09:36:06.037 INFO:teuthology.orchestra.run.smithi194.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-09T09:36:06.037 INFO:teuthology.orchestra.run.smithi194.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-09T09:36:06.040 INFO:teuthology.orchestra.run.smithi052.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-09T09:36:06.049 INFO:teuthology.orchestra.run.smithi052.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-09T09:36:06.059 INFO:teuthology.orchestra.run.smithi052.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-09T09:36:06.069 INFO:teuthology.orchestra.run.smithi052.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-09T09:36:06.090 INFO:teuthology.orchestra.run.smithi052.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-09T09:36:06.096 INFO:teuthology.orchestra.run.smithi052.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-09T09:36:06.097 INFO:teuthology.orchestra.run.smithi194.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-1742-g89fb622b-1bionic [665 kB] 2021-03-09T09:36:06.103 INFO:teuthology.orchestra.run.smithi194.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-09T09:36:06.104 INFO:teuthology.orchestra.run.smithi194.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-09T09:36:06.106 INFO:teuthology.orchestra.run.smithi194.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-09T09:36:06.107 INFO:teuthology.orchestra.run.smithi194.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-09T09:36:06.107 INFO:teuthology.orchestra.run.smithi194.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-09T09:36:06.110 INFO:teuthology.orchestra.run.smithi194.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-09T09:36:06.113 INFO:teuthology.orchestra.run.smithi194.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-09T09:36:06.116 INFO:teuthology.orchestra.run.smithi052.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-09T09:36:06.128 INFO:teuthology.orchestra.run.smithi052.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-09T09:36:06.128 INFO:teuthology.orchestra.run.smithi052.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-09T09:36:06.140 INFO:teuthology.orchestra.run.smithi194.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-1742-g89fb622b-1bionic [340 kB] 2021-03-09T09:36:06.170 INFO:teuthology.orchestra.run.smithi194.stdout:Get:30 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-1742-g89fb622b-1bionic [21.7 kB] 2021-03-09T09:36:06.171 INFO:teuthology.orchestra.run.smithi194.stdout:Get:31 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-1742-g89fb622b-1bionic [163 kB] 2021-03-09T09:36:06.177 INFO:teuthology.orchestra.run.smithi194.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-1742-g89fb622b-1bionic [29.4 kB] 2021-03-09T09:36:06.178 INFO:teuthology.orchestra.run.smithi194.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-1742-g89fb622b-1bionic [336 kB] 2021-03-09T09:36:06.181 INFO:teuthology.orchestra.run.smithi194.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-09T09:36:06.182 INFO:teuthology.orchestra.run.smithi194.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-09T09:36:06.196 INFO:teuthology.orchestra.run.smithi052.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-09T09:36:06.197 INFO:teuthology.orchestra.run.smithi052.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-09T09:36:06.199 INFO:teuthology.orchestra.run.smithi052.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-09T09:36:06.200 INFO:teuthology.orchestra.run.smithi052.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-09T09:36:06.200 INFO:teuthology.orchestra.run.smithi052.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-09T09:36:06.202 INFO:teuthology.orchestra.run.smithi194.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-09T09:36:06.203 INFO:teuthology.orchestra.run.smithi052.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-09T09:36:06.206 INFO:teuthology.orchestra.run.smithi194.stdout:Get:37 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-1742-g89fb622b-1bionic [3,311 kB] 2021-03-09T09:36:06.206 INFO:teuthology.orchestra.run.smithi052.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-09T09:36:06.260 INFO:teuthology.orchestra.run.smithi194.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-09T09:36:06.260 INFO:teuthology.orchestra.run.smithi194.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-09T09:36:06.261 INFO:teuthology.orchestra.run.smithi194.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-09T09:36:06.265 INFO:teuthology.orchestra.run.smithi194.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-09T09:36:06.273 INFO:teuthology.orchestra.run.smithi194.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-09T09:36:06.275 INFO:teuthology.orchestra.run.smithi052.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-09T09:36:06.276 INFO:teuthology.orchestra.run.smithi052.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-09T09:36:06.277 INFO:teuthology.orchestra.run.smithi194.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-09T09:36:06.278 INFO:teuthology.orchestra.run.smithi194.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-09T09:36:06.280 INFO:teuthology.orchestra.run.smithi052.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-1742-g89fb622b-1bionic [3,169 kB] 2021-03-09T09:36:06.297 INFO:teuthology.orchestra.run.smithi052.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-09T09:36:06.337 INFO:teuthology.orchestra.run.smithi194.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-09T09:36:06.355 INFO:teuthology.orchestra.run.smithi052.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-09T09:36:06.355 INFO:teuthology.orchestra.run.smithi052.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-09T09:36:06.356 INFO:teuthology.orchestra.run.smithi052.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-09T09:36:06.360 INFO:teuthology.orchestra.run.smithi052.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-09T09:36:06.368 INFO:teuthology.orchestra.run.smithi052.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-09T09:36:06.372 INFO:teuthology.orchestra.run.smithi052.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-09T09:36:06.373 INFO:teuthology.orchestra.run.smithi052.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-09T09:36:06.384 INFO:teuthology.orchestra.run.smithi194.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-1742-g89fb622b-1bionic [99.3 kB] 2021-03-09T09:36:06.387 INFO:teuthology.orchestra.run.smithi194.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-1742-g89fb622b-1bionic [414 kB] 2021-03-09T09:36:06.413 INFO:teuthology.orchestra.run.smithi194.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-1742-g89fb622b-1bionic [21.2 MB] 2021-03-09T09:36:06.426 INFO:teuthology.orchestra.run.smithi194.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-09T09:36:06.426 INFO:teuthology.orchestra.run.smithi194.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-09T09:36:06.434 INFO:teuthology.orchestra.run.smithi052.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-09T09:36:06.523 INFO:teuthology.orchestra.run.smithi052.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-09T09:36:06.524 INFO:teuthology.orchestra.run.smithi052.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-09T09:36:06.669 INFO:teuthology.orchestra.run.smithi052.stdout:Get:41 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-1742-g89fb622b-1bionic [665 kB] 2021-03-09T09:36:06.695 INFO:teuthology.orchestra.run.smithi194.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-09T09:36:06.722 INFO:teuthology.orchestra.run.smithi052.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-1742-g89fb622b-1bionic [340 kB] 2021-03-09T09:36:06.736 INFO:teuthology.orchestra.run.smithi194.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-09T09:36:06.737 INFO:teuthology.orchestra.run.smithi194.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-09T09:36:06.766 INFO:teuthology.orchestra.run.smithi052.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-1742-g89fb622b-1bionic [21.7 kB] 2021-03-09T09:36:06.767 INFO:teuthology.orchestra.run.smithi052.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-1742-g89fb622b-1bionic [163 kB] 2021-03-09T09:36:06.771 INFO:teuthology.orchestra.run.smithi052.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-1742-g89fb622b-1bionic [29.4 kB] 2021-03-09T09:36:06.772 INFO:teuthology.orchestra.run.smithi052.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-1742-g89fb622b-1bionic [336 kB] 2021-03-09T09:36:06.772 INFO:teuthology.orchestra.run.smithi194.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-09T09:36:06.773 INFO:teuthology.orchestra.run.smithi194.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-09T09:36:06.773 INFO:teuthology.orchestra.run.smithi194.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-09T09:36:06.774 INFO:teuthology.orchestra.run.smithi194.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-09T09:36:06.774 INFO:teuthology.orchestra.run.smithi194.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-09T09:36:06.775 INFO:teuthology.orchestra.run.smithi194.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-09T09:36:06.775 INFO:teuthology.orchestra.run.smithi194.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-09T09:36:06.796 INFO:teuthology.orchestra.run.smithi052.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-09T09:36:06.837 INFO:teuthology.orchestra.run.smithi052.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-09T09:36:06.838 INFO:teuthology.orchestra.run.smithi052.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-09T09:36:06.839 INFO:teuthology.orchestra.run.smithi052.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-1742-g89fb622b-1bionic [3,311 kB] 2021-03-09T09:36:06.852 INFO:teuthology.orchestra.run.smithi194.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-09T09:36:06.853 INFO:teuthology.orchestra.run.smithi194.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-09T09:36:06.859 INFO:teuthology.orchestra.run.smithi194.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-09T09:36:06.860 INFO:teuthology.orchestra.run.smithi194.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-09T09:36:06.862 INFO:teuthology.orchestra.run.smithi194.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-09T09:36:06.873 INFO:teuthology.orchestra.run.smithi052.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-09T09:36:06.874 INFO:teuthology.orchestra.run.smithi052.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-09T09:36:06.875 INFO:teuthology.orchestra.run.smithi052.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-09T09:36:06.875 INFO:teuthology.orchestra.run.smithi052.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-09T09:36:06.877 INFO:teuthology.orchestra.run.smithi052.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-09T09:36:06.878 INFO:teuthology.orchestra.run.smithi052.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-09T09:36:06.879 INFO:teuthology.orchestra.run.smithi052.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-09T09:36:06.930 INFO:teuthology.orchestra.run.smithi194.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-09T09:36:06.931 INFO:teuthology.orchestra.run.smithi194.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-09T09:36:06.934 INFO:teuthology.orchestra.run.smithi194.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-09T09:36:06.937 INFO:teuthology.orchestra.run.smithi194.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-09T09:36:06.938 INFO:teuthology.orchestra.run.smithi194.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-09T09:36:06.955 INFO:teuthology.orchestra.run.smithi052.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-09T09:36:06.957 INFO:teuthology.orchestra.run.smithi052.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-09T09:36:06.965 INFO:teuthology.orchestra.run.smithi052.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-09T09:36:06.965 INFO:teuthology.orchestra.run.smithi052.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-09T09:36:07.008 INFO:teuthology.orchestra.run.smithi194.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-09T09:36:07.013 INFO:teuthology.orchestra.run.smithi194.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-09T09:36:07.022 INFO:teuthology.orchestra.run.smithi194.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-09T09:36:07.024 INFO:teuthology.orchestra.run.smithi194.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-09T09:36:07.025 INFO:teuthology.orchestra.run.smithi194.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-09T09:36:07.034 INFO:teuthology.orchestra.run.smithi052.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-09T09:36:07.035 INFO:teuthology.orchestra.run.smithi052.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-09T09:36:07.036 INFO:teuthology.orchestra.run.smithi052.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-09T09:36:07.041 INFO:teuthology.orchestra.run.smithi052.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-09T09:36:07.044 INFO:teuthology.orchestra.run.smithi052.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-09T09:36:07.046 INFO:teuthology.orchestra.run.smithi052.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-09T09:36:07.086 INFO:teuthology.orchestra.run.smithi194.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-09T09:36:07.087 INFO:teuthology.orchestra.run.smithi194.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-09T09:36:07.088 INFO:teuthology.orchestra.run.smithi194.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-09T09:36:07.092 INFO:teuthology.orchestra.run.smithi194.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-09T09:36:07.093 INFO:teuthology.orchestra.run.smithi194.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-09T09:36:07.113 INFO:teuthology.orchestra.run.smithi052.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-09T09:36:07.118 INFO:teuthology.orchestra.run.smithi052.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-09T09:36:07.126 INFO:teuthology.orchestra.run.smithi052.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-09T09:36:07.128 INFO:teuthology.orchestra.run.smithi052.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-09T09:36:07.164 INFO:teuthology.orchestra.run.smithi194.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-09T09:36:07.165 INFO:teuthology.orchestra.run.smithi194.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-09T09:36:07.167 INFO:teuthology.orchestra.run.smithi194.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-09T09:36:07.188 INFO:teuthology.orchestra.run.smithi194.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-09T09:36:07.189 INFO:teuthology.orchestra.run.smithi194.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-09T09:36:07.193 INFO:teuthology.orchestra.run.smithi052.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-09T09:36:07.194 INFO:teuthology.orchestra.run.smithi052.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-09T09:36:07.195 INFO:teuthology.orchestra.run.smithi052.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-09T09:36:07.197 INFO:teuthology.orchestra.run.smithi052.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-09T09:36:07.201 INFO:teuthology.orchestra.run.smithi052.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-09T09:36:07.202 INFO:teuthology.orchestra.run.smithi052.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-09T09:36:07.242 INFO:teuthology.orchestra.run.smithi194.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-09T09:36:07.259 INFO:teuthology.orchestra.run.smithi052.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-1742-g89fb622b-1bionic [99.3 kB] 2021-03-09T09:36:07.262 INFO:teuthology.orchestra.run.smithi052.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-1742-g89fb622b-1bionic [414 kB] 2021-03-09T09:36:07.272 INFO:teuthology.orchestra.run.smithi052.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-09T09:36:07.273 INFO:teuthology.orchestra.run.smithi052.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-09T09:36:07.276 INFO:teuthology.orchestra.run.smithi052.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-09T09:36:07.296 INFO:teuthology.orchestra.run.smithi052.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-09T09:36:07.332 INFO:teuthology.orchestra.run.smithi052.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-1742-g89fb622b-1bionic [21.2 MB] 2021-03-09T09:36:07.339 INFO:teuthology.orchestra.run.smithi194.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-09T09:36:07.340 INFO:teuthology.orchestra.run.smithi194.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-09T09:36:07.341 INFO:teuthology.orchestra.run.smithi194.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-03-09T09:36:07.344 INFO:teuthology.orchestra.run.smithi194.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-09T09:36:07.345 INFO:teuthology.orchestra.run.smithi194.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-09T09:36:07.345 INFO:teuthology.orchestra.run.smithi194.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-09T09:36:07.346 INFO:teuthology.orchestra.run.smithi194.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-09T09:36:07.351 INFO:teuthology.orchestra.run.smithi052.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-09T09:36:07.353 INFO:teuthology.orchestra.run.smithi052.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-09T09:36:07.356 INFO:teuthology.orchestra.run.smithi194.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-09T09:36:07.356 INFO:teuthology.orchestra.run.smithi194.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-09T09:36:07.368 INFO:teuthology.orchestra.run.smithi194.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-1742-g89fb622b-1bionic [5,646 kB] 2021-03-09T09:36:07.454 INFO:teuthology.orchestra.run.smithi052.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-09T09:36:07.454 INFO:teuthology.orchestra.run.smithi052.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-09T09:36:07.455 INFO:teuthology.orchestra.run.smithi052.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-03-09T09:36:07.459 INFO:teuthology.orchestra.run.smithi052.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-09T09:36:07.459 INFO:teuthology.orchestra.run.smithi052.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-09T09:36:07.460 INFO:teuthology.orchestra.run.smithi052.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-09T09:36:07.460 INFO:teuthology.orchestra.run.smithi052.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-09T09:36:07.462 INFO:teuthology.orchestra.run.smithi052.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-09T09:36:07.510 INFO:teuthology.orchestra.run.smithi052.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-09T09:36:07.598 INFO:teuthology.orchestra.run.smithi194.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-1742-g89fb622b-1bionic [184 kB] 2021-03-09T09:36:07.739 INFO:teuthology.orchestra.run.smithi194.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-1742-g89fb622b-1bionic [1,388 kB] 2021-03-09T09:36:07.760 INFO:teuthology.orchestra.run.smithi194.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-1742-g89fb622b-1bionic [6,670 kB] 2021-03-09T09:36:07.998 INFO:teuthology.orchestra.run.smithi194.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-1742-g89fb622b-1bionic [24.9 MB] 2021-03-09T09:36:09.135 INFO:teuthology.orchestra.run.smithi194.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-1742-g89fb622b-1bionic [3,812 B] 2021-03-09T09:36:09.136 INFO:teuthology.orchestra.run.smithi194.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-1742-g89fb622b-1bionic [771 kB] 2021-03-09T09:36:09.157 INFO:teuthology.orchestra.run.smithi194.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-1742-g89fb622b-1bionic [2,153 kB] 2021-03-09T09:36:09.253 INFO:teuthology.orchestra.run.smithi194.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-1742-g89fb622b-1bionic [63.1 kB] 2021-03-09T09:36:09.255 INFO:teuthology.orchestra.run.smithi194.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-1742-g89fb622b-1bionic [68.9 kB] 2021-03-09T09:36:09.256 INFO:teuthology.orchestra.run.smithi194.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-1742-g89fb622b-1bionic [2,333 kB] 2021-03-09T09:36:09.375 INFO:teuthology.orchestra.run.smithi194.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-1742-g89fb622b-1bionic [7,881 kB] 2021-03-09T09:36:09.732 INFO:teuthology.orchestra.run.smithi194.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-1742-g89fb622b-1bionic [3,928 B] 2021-03-09T09:36:09.732 INFO:teuthology.orchestra.run.smithi194.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-1742-g89fb622b-1bionic [22.5 kB] 2021-03-09T09:36:09.732 INFO:teuthology.orchestra.run.smithi194.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-1742-g89fb622b-1bionic [45.2 MB] 2021-03-09T09:36:09.782 INFO:teuthology.orchestra.run.smithi052.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-1742-g89fb622b-1bionic [5,646 kB] 2021-03-09T09:36:10.378 INFO:teuthology.orchestra.run.smithi052.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-1742-g89fb622b-1bionic [184 kB] 2021-03-09T09:36:10.383 INFO:teuthology.orchestra.run.smithi052.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-1742-g89fb622b-1bionic [1,388 kB] 2021-03-09T09:36:10.518 INFO:teuthology.orchestra.run.smithi052.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-1742-g89fb622b-1bionic [6,670 kB] 2021-03-09T09:36:11.099 INFO:teuthology.orchestra.run.smithi052.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-1742-g89fb622b-1bionic [24.9 MB] 2021-03-09T09:36:11.991 INFO:teuthology.orchestra.run.smithi194.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-1742-g89fb622b-1bionic [19.9 kB] 2021-03-09T09:36:11.992 INFO:teuthology.orchestra.run.smithi194.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-1742-g89fb622b-1bionic [10.2 MB] 2021-03-09T09:36:12.486 INFO:teuthology.orchestra.run.smithi194.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-1742-g89fb622b-1bionic [77.7 kB] 2021-03-09T09:36:12.827 INFO:teuthology.orchestra.run.smithi052.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-1742-g89fb622b-1bionic [3,812 B] 2021-03-09T09:36:12.827 INFO:teuthology.orchestra.run.smithi052.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-1742-g89fb622b-1bionic [771 kB] 2021-03-09T09:36:12.849 INFO:teuthology.orchestra.run.smithi052.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-1742-g89fb622b-1bionic [2,153 kB] 2021-03-09T09:36:12.920 INFO:teuthology.orchestra.run.smithi052.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-1742-g89fb622b-1bionic [63.1 kB] 2021-03-09T09:36:12.921 INFO:teuthology.orchestra.run.smithi052.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-1742-g89fb622b-1bionic [68.9 kB] 2021-03-09T09:36:12.924 INFO:teuthology.orchestra.run.smithi052.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-1742-g89fb622b-1bionic [2,333 kB] 2021-03-09T09:36:13.001 INFO:teuthology.orchestra.run.smithi052.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-1742-g89fb622b-1bionic [7,881 kB] 2021-03-09T09:36:13.185 INFO:teuthology.orchestra.run.smithi194.stdout:Fetched 166 MB in 7s (23.2 MB/s) 2021-03-09T09:36:13.260 INFO:teuthology.orchestra.run.smithi052.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-1742-g89fb622b-1bionic [3,928 B] 2021-03-09T09:36:13.261 INFO:teuthology.orchestra.run.smithi052.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-1742-g89fb622b-1bionic [22.5 kB] 2021-03-09T09:36:13.261 INFO:teuthology.orchestra.run.smithi052.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-1742-g89fb622b-1bionic [45.2 MB] 2021-03-09T09:36:13.306 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-09T09:36:13.341 INFO:teuthology.orchestra.run.smithi194.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-09T09:36:13.344 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-09T09:36:13.388 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T09:36:13.607 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libjaeger. 2021-03-09T09:36:13.618 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../001-libjaeger_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:13.649 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:13.951 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-09T09:36:13.963 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-09T09:36:14.060 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T09:36:14.391 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../003-librbd1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:14.513 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking librbd1 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T09:36:14.746 INFO:teuthology.orchestra.run.smithi052.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-1742-g89fb622b-1bionic [19.9 kB] 2021-03-09T09:36:14.746 INFO:teuthology.orchestra.run.smithi052.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-1742-g89fb622b-1bionic [10.2 MB] 2021-03-09T09:36:15.091 INFO:teuthology.orchestra.run.smithi052.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-1742-g89fb622b-1bionic [77.7 kB] 2021-03-09T09:36:15.153 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../004-librados2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:15.251 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking librados2 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T09:36:15.808 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 166 MB in 10s (17.1 MB/s) 2021-03-09T09:36:15.895 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libcephfs2. 2021-03-09T09:36:15.907 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:15.945 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:15.964 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-09T09:36:16.003 INFO:teuthology.orchestra.run.smithi052.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-09T09:36:16.006 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-09T09:36:16.037 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T09:36:16.264 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libjaeger. 2021-03-09T09:36:16.273 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-rados. 2021-03-09T09:36:16.276 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../001-libjaeger_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:16.283 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../006-python3-rados_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:16.315 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:16.323 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:16.633 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-09T09:36:16.645 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-09T09:36:16.751 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T09:36:16.758 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-09T09:36:16.770 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:16.800 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:17.061 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-cephfs. 2021-03-09T09:36:17.073 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:17.083 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../003-librbd1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:17.111 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:17.313 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librbd1 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T09:36:17.430 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-09T09:36:17.441 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:17.480 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:17.799 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-prettytable. 2021-03-09T09:36:17.809 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-03-09T09:36:17.849 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-09T09:36:17.944 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../004-librados2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:18.066 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librados2 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T09:36:18.093 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-rbd. 2021-03-09T09:36:18.104 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:18.143 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:18.454 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-09T09:36:18.466 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-09T09:36:18.504 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T09:36:18.712 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libcephfs2. 2021-03-09T09:36:18.724 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:18.761 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:18.865 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-09T09:36:18.876 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-09T09:36:18.915 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T09:36:19.080 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rados. 2021-03-09T09:36:19.092 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../006-python3-rados_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:19.130 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:19.250 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package librgw2. 2021-03-09T09:36:19.263 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../014-librgw2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:19.301 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:19.458 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-09T09:36:19.469 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:19.499 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:19.768 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cephfs. 2021-03-09T09:36:19.779 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:19.818 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:19.853 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-rgw. 2021-03-09T09:36:19.863 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:19.894 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:20.121 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-09T09:36:20.133 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:20.196 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:20.196 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-09T09:36:20.209 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-09T09:36:20.255 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T09:36:20.507 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-prettytable. 2021-03-09T09:36:20.519 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-03-09T09:36:20.557 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-09T09:36:20.557 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-09T09:36:20.568 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-09T09:36:20.608 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T09:36:20.793 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rbd. 2021-03-09T09:36:20.803 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:20.834 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:20.985 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package liboath0. 2021-03-09T09:36:20.998 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-03-09T09:36:21.035 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-09T09:36:21.212 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-09T09:36:21.224 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-09T09:36:21.261 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T09:36:21.321 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libradosstriper1. 2021-03-09T09:36:21.334 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:21.363 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:21.581 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-09T09:36:21.593 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-09T09:36:21.631 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T09:36:21.966 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-common. 2021-03-09T09:36:21.975 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package librgw2. 2021-03-09T09:36:21.977 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../020-ceph-common_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:21.987 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../014-librgw2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:22.015 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:22.024 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:22.560 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rgw. 2021-03-09T09:36:22.572 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:22.602 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:22.946 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-09T09:36:22.957 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-09T09:36:22.996 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T09:36:23.299 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-09T09:36:23.311 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-09T09:36:23.348 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T09:36:23.818 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package liboath0. 2021-03-09T09:36:23.830 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-03-09T09:36:23.868 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-09T09:36:23.904 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-base. 2021-03-09T09:36:23.916 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../021-ceph-base_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:23.963 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:24.145 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libradosstriper1. 2021-03-09T09:36:24.157 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:24.195 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:24.673 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-common. 2021-03-09T09:36:24.685 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../020-ceph-common_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:24.723 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:24.900 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-dateutil. 2021-03-09T09:36:24.913 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-03-09T09:36:24.949 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-09T09:36:25.260 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-09T09:36:25.273 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-09T09:36:25.310 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-09T09:36:25.604 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-09T09:36:25.617 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-09T09:36:25.654 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-09T09:36:25.949 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-cryptography. 2021-03-09T09:36:25.961 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-09T09:36:25.990 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T09:36:26.343 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-openssl. 2021-03-09T09:36:26.356 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-09T09:36:26.385 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T09:36:26.570 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-base. 2021-03-09T09:36:26.581 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../021-ceph-base_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:26.620 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:26.646 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-09T09:36:26.658 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:26.696 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:27.025 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-09T09:36:27.037 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-09T09:36:27.075 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-09T09:36:27.327 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-09T09:36:27.340 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-09T09:36:27.377 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-09T09:36:27.549 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-dateutil. 2021-03-09T09:36:27.561 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-03-09T09:36:27.599 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-09T09:36:27.705 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-jwt. 2021-03-09T09:36:27.718 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-09T09:36:27.747 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-09T09:36:27.935 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-09T09:36:27.947 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-09T09:36:28.010 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-09T09:36:28.058 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-logutils. 2021-03-09T09:36:28.069 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-03-09T09:36:28.108 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-09T09:36:28.296 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-09T09:36:28.308 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-09T09:36:28.346 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-09T09:36:28.411 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-09T09:36:28.421 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-09T09:36:28.460 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-09T09:36:28.606 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cryptography. 2021-03-09T09:36:28.619 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-09T09:36:28.648 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T09:36:28.763 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-mako. 2021-03-09T09:36:28.775 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-09T09:36:28.813 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-09T09:36:29.001 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-openssl. 2021-03-09T09:36:29.013 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-09T09:36:29.043 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T09:36:29.141 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-09T09:36:29.152 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-09T09:36:29.182 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-09T09:36:29.412 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-09T09:36:29.424 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:29.460 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-09T09:36:29.462 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:29.473 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-09T09:36:29.510 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-09T09:36:29.812 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-webob. 2021-03-09T09:36:29.824 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-09T09:36:29.841 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-09T09:36:29.854 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-09T09:36:29.862 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T09:36:29.891 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-09T09:36:30.182 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-bs4. 2021-03-09T09:36:30.193 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-03-09T09:36:30.202 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-09T09:36:30.214 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-09T09:36:30.231 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-09T09:36:30.252 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-09T09:36:30.567 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-waitress. 2021-03-09T09:36:30.579 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-03-09T09:36:30.622 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-jwt. 2021-03-09T09:36:30.633 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-09T09:36:30.634 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-09T09:36:30.671 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-09T09:36:30.933 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-logutils. 2021-03-09T09:36:30.945 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-03-09T09:36:30.962 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-tempita. 2021-03-09T09:36:30.973 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-03-09T09:36:30.982 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-09T09:36:31.012 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-09T09:36:31.273 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-paste. 2021-03-09T09:36:31.284 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-09T09:36:31.294 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-09T09:36:31.305 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-09T09:36:31.314 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T09:36:31.344 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-09T09:36:31.641 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-09T09:36:31.652 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-09T09:36:31.671 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-mako. 2021-03-09T09:36:31.683 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-09T09:36:31.690 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T09:36:31.721 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-09T09:36:31.960 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-09T09:36:31.972 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-09T09:36:32.001 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-09T09:36:32.032 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-09T09:36:32.045 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-09T09:36:32.082 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-09T09:36:32.337 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-webtest. 2021-03-09T09:36:32.349 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-09T09:36:32.387 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T09:36:32.401 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-09T09:36:32.412 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-09T09:36:32.460 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-09T09:36:32.715 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pecan. 2021-03-09T09:36:32.727 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-03-09T09:36:32.737 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-webob. 2021-03-09T09:36:32.748 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-09T09:36:32.765 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-09T09:36:32.845 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T09:36:33.103 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-09T09:36:33.116 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-09T09:36:33.152 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T09:36:33.340 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-bs4. 2021-03-09T09:36:33.353 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-03-09T09:36:33.389 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-09T09:36:33.468 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr. 2021-03-09T09:36:33.479 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:33.510 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:33.684 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-waitress. 2021-03-09T09:36:33.697 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-03-09T09:36:33.733 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-09T09:36:33.979 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mon. 2021-03-09T09:36:33.991 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:34.029 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:34.086 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-tempita. 2021-03-09T09:36:34.097 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-03-09T09:36:34.136 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-09T09:36:34.438 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-paste. 2021-03-09T09:36:34.450 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-09T09:36:34.488 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T09:36:34.815 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-09T09:36:34.828 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-09T09:36:34.865 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T09:36:34.983 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-osd. 2021-03-09T09:36:34.995 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:35.032 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:35.168 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-09T09:36:35.181 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-09T09:36:35.218 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-09T09:36:35.470 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-webtest. 2021-03-09T09:36:35.482 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-09T09:36:35.512 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T09:36:35.865 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pecan. 2021-03-09T09:36:35.876 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-03-09T09:36:35.915 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-09T09:36:36.266 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-09T09:36:36.277 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-09T09:36:36.316 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T09:36:36.694 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr. 2021-03-09T09:36:36.707 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:36.743 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:37.155 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mon. 2021-03-09T09:36:37.168 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:37.222 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:37.704 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph. 2021-03-09T09:36:37.716 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../049-ceph_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:37.754 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:38.056 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-fuse. 2021-03-09T09:36:38.069 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:38.106 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:38.493 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mds. 2021-03-09T09:36:38.504 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:38.535 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:38.859 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-osd. 2021-03-09T09:36:38.871 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:38.908 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:39.046 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package cephadm. 2021-03-09T09:36:39.059 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../052-cephadm_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:39.096 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:39.415 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-jinja2. 2021-03-09T09:36:39.427 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-09T09:36:39.465 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T09:36:39.810 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-09T09:36:39.823 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:39.860 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:40.171 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-09T09:36:40.184 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-03-09T09:36:40.221 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-09T09:36:40.507 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-routes. 2021-03-09T09:36:40.520 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-03-09T09:36:40.557 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-09T09:36:40.834 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-09T09:36:40.847 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:40.876 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:40.906 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph. 2021-03-09T09:36:40.917 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../049-ceph_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:40.956 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:41.300 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-fuse. 2021-03-09T09:36:41.312 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:41.350 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:41.761 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mds. 2021-03-09T09:36:41.883 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:41.883 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:41.884 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-numpy. 2021-03-09T09:36:41.884 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-09T09:36:41.884 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T09:36:42.264 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package cephadm. 2021-03-09T09:36:42.275 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../052-cephadm_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:42.314 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:42.433 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-decorator. 2021-03-09T09:36:42.446 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-09T09:36:42.483 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-09T09:36:42.625 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-jinja2. 2021-03-09T09:36:42.636 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-09T09:36:42.675 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T09:36:42.844 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-scipy. 2021-03-09T09:36:42.857 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-09T09:36:42.902 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T09:36:43.020 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-09T09:36:43.033 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:43.069 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:43.389 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-09T09:36:43.402 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-03-09T09:36:43.439 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-09T09:36:43.767 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-routes. 2021-03-09T09:36:43.779 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-03-09T09:36:43.817 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-09T09:36:44.009 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-09T09:36:44.023 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-09T09:36:44.060 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-09T09:36:44.130 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-09T09:36:44.142 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:44.180 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:44.512 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-joblib. 2021-03-09T09:36:44.525 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-09T09:36:44.562 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-09T09:36:44.899 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-sklearn. 2021-03-09T09:36:44.912 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-09T09:36:44.957 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-09T09:36:45.079 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-numpy. 2021-03-09T09:36:45.094 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-09T09:36:45.137 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T09:36:45.471 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-09T09:36:45.484 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:45.521 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:45.776 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-decorator. 2021-03-09T09:36:45.790 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-09T09:36:45.826 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-09T09:36:46.137 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-scipy. 2021-03-09T09:36:46.149 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-09T09:36:46.187 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T09:36:46.599 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-monotonic. 2021-03-09T09:36:46.613 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-09T09:36:46.658 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-09T09:36:46.960 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-fasteners. 2021-03-09T09:36:46.973 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-09T09:36:47.010 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-09T09:36:47.364 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-httplib2. 2021-03-09T09:36:47.371 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-09T09:36:47.376 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-09T09:36:47.384 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-09T09:36:47.413 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T09:36:47.421 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-09T09:36:47.699 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-09T09:36:47.713 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-09T09:36:47.757 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-09T09:36:47.899 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-joblib. 2021-03-09T09:36:47.911 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-09T09:36:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-09T09:36:48.094 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-09T09:36:48.106 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-09T09:36:48.144 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T09:36:48.377 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-sklearn. 2021-03-09T09:36:48.389 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-09T09:36:48.427 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-09T09:36:48.455 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-rsa. 2021-03-09T09:36:48.468 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-09T09:36:48.505 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-09T09:36:48.833 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-09T09:36:48.844 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-09T09:36:48.882 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T09:36:48.975 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-09T09:36:48.986 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:49.025 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:49.328 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-09T09:36:49.340 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-09T09:36:49.387 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-09T09:36:49.715 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-setuptools. 2021-03-09T09:36:49.727 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-09T09:36:49.765 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-09T09:36:50.145 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-monotonic. 2021-03-09T09:36:50.151 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-websocket. 2021-03-09T09:36:50.158 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-09T09:36:50.164 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-09T09:36:50.195 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-09T09:36:50.209 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T09:36:50.520 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-09T09:36:50.522 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-fasteners. 2021-03-09T09:36:50.534 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-09T09:36:50.535 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-09T09:36:50.572 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-09T09:36:50.579 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T09:36:50.900 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-httplib2. 2021-03-09T09:36:50.912 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-09T09:36:50.950 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T09:36:51.024 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-09T09:36:51.036 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:51.074 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:51.219 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-09T09:36:51.233 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-09T09:36:51.270 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-09T09:36:51.393 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-09T09:36:51.406 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-09T09:36:51.443 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-09T09:36:51.615 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-09T09:36:51.627 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-09T09:36:51.665 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T09:36:51.821 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-09T09:36:51.833 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-09T09:36:51.866 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T09:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rsa. 2021-03-09T09:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-09T09:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-09T09:36:52.140 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-09T09:36:52.154 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:52.199 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-09T09:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-09T09:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T09:36:52.735 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-09T09:36:52.748 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-09T09:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-09T09:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-09T09:36:52.862 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-09T09:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-09T09:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-setuptools. 2021-03-09T09:36:53.214 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-09T09:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-09T09:36:53.229 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-09T09:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-09T09:36:53.264 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T09:36:53.525 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package jq. 2021-03-09T09:36:53.539 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-09T09:36:53.576 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-09T09:36:53.614 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-websocket. 2021-03-09T09:36:53.627 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-09T09:36:53.664 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T09:36:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package socat. 2021-03-09T09:36:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-09T09:36:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-09T09:36:53.959 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-09T09:36:53.973 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-09T09:36:54.017 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T09:36:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-09T09:36:54.260 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-09T09:36:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T09:36:54.440 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-09T09:36:54.454 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:54.498 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:54.550 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package xmlstarlet. 2021-03-09T09:36:54.563 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-09T09:36:54.603 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-09T09:36:54.792 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-09T09:36:54.805 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-09T09:36:54.842 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-09T09:36:54.971 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-test. 2021-03-09T09:36:54.983 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../086-ceph-test_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:55.021 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:55.221 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-09T09:36:55.233 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-09T09:36:55.273 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T09:36:55.573 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-09T09:36:55.580 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:36:55.623 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:55.935 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-09T09:36:55.947 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-09T09:36:56.070 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-09T09:36:56.415 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-09T09:36:56.427 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-09T09:36:56.465 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T09:36:56.709 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package jq. 2021-03-09T09:36:56.722 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-09T09:36:56.776 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-09T09:36:57.020 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package socat. 2021-03-09T09:36:57.031 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-09T09:36:57.070 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-09T09:36:57.449 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-09T09:36:57.461 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-09T09:36:57.498 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T09:36:57.801 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package xmlstarlet. 2021-03-09T09:36:57.815 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-09T09:36:57.859 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-09T09:36:58.197 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-test. 2021-03-09T09:36:58.209 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../086-ceph-test_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:58.247 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:58.416 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-09T09:36:58.428 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:36:58.466 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:36:58.743 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package nvme-cli. 2021-03-09T09:36:58.754 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-09T09:36:58.793 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T09:36:59.131 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python-attr. 2021-03-09T09:36:59.144 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-09T09:36:59.181 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-09T09:36:59.500 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python-funcsigs. 2021-03-09T09:36:59.512 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-09T09:36:59.550 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-09T09:36:59.853 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python-pluggy. 2021-03-09T09:36:59.866 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-09T09:36:59.903 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-09T09:37:00.197 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python-py. 2021-03-09T09:37:00.210 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-09T09:37:00.247 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-09T09:37:00.567 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python-pytest. 2021-03-09T09:37:00.579 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-09T09:37:00.616 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-09T09:37:00.987 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-attr. 2021-03-09T09:37:01.000 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-09T09:37:01.045 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-09T09:37:01.339 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-cycler. 2021-03-09T09:37:01.352 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-09T09:37:01.389 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-09T09:37:01.617 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-09T09:37:01.632 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:37:01.675 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:01.692 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-webencodings. 2021-03-09T09:37:01.704 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-09T09:37:01.742 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-09T09:37:01.978 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package nvme-cli. 2021-03-09T09:37:01.986 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-html5lib. 2021-03-09T09:37:01.991 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-09T09:37:01.997 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-09T09:37:02.020 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T09:37:02.027 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-09T09:37:02.358 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-attr. 2021-03-09T09:37:02.369 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-09T09:37:02.406 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-09T09:37:02.408 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-09T09:37:02.412 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-09T09:37:02.455 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T09:37:02.711 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-funcsigs. 2021-03-09T09:37:02.724 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-09T09:37:02.761 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-09T09:37:02.883 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-09T09:37:02.897 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-09T09:37:02.942 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T09:37:03.072 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-pluggy. 2021-03-09T09:37:03.085 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-09T09:37:03.130 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-09T09:37:03.269 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-tz. 2021-03-09T09:37:03.283 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-09T09:37:03.328 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-09T09:37:03.392 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-py. 2021-03-09T09:37:03.403 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-09T09:37:03.441 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-09T09:37:03.589 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-09T09:37:03.603 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-09T09:37:03.639 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T09:37:04.044 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-pytest. 2021-03-09T09:37:04.057 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-09T09:37:04.094 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-09T09:37:04.277 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-olefile. 2021-03-09T09:37:04.289 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-09T09:37:04.327 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-09T09:37:04.406 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-attr. 2021-03-09T09:37:04.419 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-09T09:37:04.456 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-09T09:37:04.588 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pastescript. 2021-03-09T09:37:04.600 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-09T09:37:04.630 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-09T09:37:04.784 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cycler. 2021-03-09T09:37:04.796 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-09T09:37:04.833 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-09T09:37:04.933 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-09T09:37:04.946 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-03-09T09:37:04.983 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T09:37:05.128 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-webencodings. 2021-03-09T09:37:05.140 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-09T09:37:05.178 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-09T09:37:05.361 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pluggy. 2021-03-09T09:37:05.374 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-09T09:37:05.411 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-09T09:37:05.472 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-html5lib. 2021-03-09T09:37:05.485 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-09T09:37:05.548 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-09T09:37:05.655 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-py. 2021-03-09T09:37:05.668 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-09T09:37:05.696 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-09T09:37:05.834 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-09T09:37:05.848 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-09T09:37:05.892 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T09:37:05.950 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-09T09:37:05.961 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-09T09:37:05.999 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-09T09:37:06.310 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pytest. 2021-03-09T09:37:06.324 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-09T09:37:06.329 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-09T09:37:06.341 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-09T09:37:06.368 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-09T09:37:06.404 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T09:37:06.714 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-simplejson. 2021-03-09T09:37:06.726 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-09T09:37:06.763 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-09T09:37:06.773 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-tz. 2021-03-09T09:37:06.787 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-09T09:37:06.823 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-09T09:37:07.083 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-09T09:37:07.097 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-09T09:37:07.118 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-09T09:37:07.131 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-09T09:37:07.141 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T09:37:07.159 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T09:37:07.712 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package radosgw. 2021-03-09T09:37:07.722 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../111-radosgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:37:07.761 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:07.824 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-olefile. 2021-03-09T09:37:07.836 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-09T09:37:07.875 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-09T09:37:08.210 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pastescript. 2021-03-09T09:37:08.221 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-09T09:37:08.259 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-09T09:37:08.621 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-09T09:37:08.633 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-03-09T09:37:08.663 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T09:37:08.773 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package rbd-fuse. 2021-03-09T09:37:08.786 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:37:08.823 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:09.058 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pluggy. 2021-03-09T09:37:09.071 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-09T09:37:09.108 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-09T09:37:09.114 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-09T09:37:09.419 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-py. 2021-03-09T09:37:09.432 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-09T09:37:09.435 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T09:37:09.469 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-09T09:37:09.789 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-09T09:37:09.801 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-09T09:37:09.837 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T09:37:09.872 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-09T09:37:09.946 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T09:37:10.166 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pytest. 2021-03-09T09:37:10.179 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-09T09:37:10.219 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-09T09:37:10.503 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-simplejson. 2021-03-09T09:37:10.517 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-09T09:37:10.540 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T09:37:10.553 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-09T09:37:10.872 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-09T09:37:10.886 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-09T09:37:10.925 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-09T09:37:10.930 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T09:37:11.201 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-09T09:37:11.368 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package radosgw. 2021-03-09T09:37:11.382 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../111-radosgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:37:11.426 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:11.510 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-09T09:37:11.711 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-09T09:37:11.821 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-09T09:37:12.122 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-09T09:37:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-09T09:37:12.479 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package rbd-fuse. 2021-03-09T09:37:12.492 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:37:12.529 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:12.766 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-09T09:37:12.806 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-09T09:37:13.109 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:13.118 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T09:37:13.418 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-09T09:37:13.494 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T09:37:13.620 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T09:37:13.720 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-09T09:37:13.821 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T09:37:14.223 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T09:37:14.599 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-09T09:37:14.725 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T09:37:14.876 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-09T09:37:15.033 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-09T09:37:15.160 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-09T09:37:15.308 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python-py (1.5.2-1) ... 2021-03-09T09:37:15.361 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-09T09:37:15.487 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-09T09:37:15.543 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-09T09:37:15.846 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-09T09:37:15.886 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-09T09:37:15.975 INFO:teuthology.orchestra.run.smithi194.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-09T09:37:16.212 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-09T09:37:16.233 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-09T09:37:16.313 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-09T09:37:16.481 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-09T09:37:16.558 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-09T09:37:16.790 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T09:37:16.859 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:17.116 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-09T09:37:17.160 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-09T09:37:17.425 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-09T09:37:17.470 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-09T09:37:17.621 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T09:37:17.793 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-09T09:37:18.119 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-09T09:37:18.404 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:18.582 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T09:37:18.917 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-09T09:37:19.029 INFO:teuthology.orchestra.run.smithi194.stdout:Adding system user cephadm....done 2021-03-09T09:37:19.193 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-09T09:37:19.218 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-py (1.5.2-1) ... 2021-03-09T09:37:19.452 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-09T09:37:19.494 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T09:37:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T09:37:19.746 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-09T09:37:19.787 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-09T09:37:19.876 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-09T09:37:20.072 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-09T09:37:20.122 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-09T09:37:20.273 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-09T09:37:20.356 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T09:37:20.466 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-09T09:37:20.482 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-09T09:37:20.767 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T09:37:20.825 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:21.101 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-09T09:37:21.143 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-09T09:37:21.386 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-09T09:37:21.444 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T09:37:21.570 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T09:37:21.762 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-09T09:37:21.913 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-09T09:37:22.088 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-09T09:37:22.239 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T09:37:22.312 INFO:teuthology.orchestra.run.smithi194.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-09T09:37:22.344 INFO:teuthology.orchestra.run.smithi194.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-09T09:37:22.381 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:22.583 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-09T09:37:22.883 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T09:37:23.065 INFO:teuthology.orchestra.run.smithi052.stdout:Adding system user cephadm....done 2021-03-09T09:37:23.221 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-09T09:37:23.409 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T09:37:23.514 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T09:37:23.664 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T09:37:23.744 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-09T09:37:23.783 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-09T09:37:24.053 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-09T09:37:24.134 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-09T09:37:24.246 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:24.347 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T09:37:24.435 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T09:37:24.586 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-09T09:37:24.723 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T09:37:24.829 INFO:teuthology.orchestra.run.smithi194.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-09T09:37:24.921 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:25.083 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-09T09:37:25.239 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-09T09:37:25.259 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T09:37:25.385 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T09:37:25.523 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T09:37:25.624 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T09:37:25.636 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T09:37:25.747 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-09T09:37:25.967 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-09T09:37:26.124 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T09:37:26.250 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-09T09:37:26.327 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T09:37:26.433 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-09T09:37:26.457 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-09T09:37:26.551 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-09T09:37:26.712 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-09T09:37:26.856 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T09:37:27.022 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T09:37:27.144 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-09T09:37:27.331 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T09:37:27.454 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-09T09:37:27.682 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-09T09:37:27.797 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-09T09:37:28.000 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-09T09:37:28.081 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-09T09:37:28.201 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:28.327 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T09:37:28.365 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-09T09:37:28.692 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T09:37:28.745 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T09:37:28.860 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-09T09:37:29.060 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-09T09:37:29.122 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-09T09:37:29.307 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T09:37:29.369 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-09T09:37:29.408 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T09:37:29.511 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T09:37:29.667 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T09:37:29.827 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-09T09:37:30.220 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T09:37:30.346 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-09T09:37:30.638 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-09T09:37:30.667 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T09:37:30.948 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T09:37:30.993 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-09T09:37:31.249 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-09T09:37:31.302 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-09T09:37:31.428 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-09T09:37:31.542 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-09T09:37:31.788 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-09T09:37:31.910 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-09T09:37:32.031 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T09:37:32.131 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-09T09:37:32.211 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-09T09:37:32.249 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up librados2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:32.375 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-09T09:37:32.529 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-09T09:37:32.701 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:32.827 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:32.855 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T09:37:32.953 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:33.078 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T09:37:33.224 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-09T09:37:33.396 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-09T09:37:33.500 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-09T09:37:33.626 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T09:37:33.681 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-09T09:37:34.133 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T09:37:34.831 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T09:37:34.994 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-09T09:37:35.166 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-09T09:37:35.475 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-09T09:37:35.601 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-09T09:37:35.824 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T09:37:35.977 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-09T09:37:36.125 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:36.245 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T09:37:36.250 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:36.371 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-09T09:37:36.376 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:36.497 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librados2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:36.502 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:36.640 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-09T09:37:36.712 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T09:37:36.712 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T09:37:36.967 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:36.979 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up librbd1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:37.077 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:37.079 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-09T09:37:37.160 INFO:teuthology.orchestra.run.smithi194.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-09T09:37:37.194 INFO:teuthology.orchestra.run.smithi194.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-09T09:37:37.201 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:37.327 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T09:37:37.475 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T09:37:37.645 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-09T09:37:37.963 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-09T09:37:38.127 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:38.253 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-09T09:37:38.374 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T09:37:38.587 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:38.688 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:38.797 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:38.898 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:39.244 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-09T09:37:39.415 INFO:teuthology.orchestra.run.smithi194.stdout:Adding group ceph....done 2021-03-09T09:37:39.793 INFO:teuthology.orchestra.run.smithi194.stdout:Adding system user ceph....done 2021-03-09T09:37:39.985 INFO:teuthology.orchestra.run.smithi194.stdout:Setting system user ceph properties....done 2021-03-09T09:37:40.115 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T09:37:40.144 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-09T09:37:40.354 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-09T09:37:40.392 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:40.501 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:40.616 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:40.627 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:40.753 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:40.865 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T09:37:40.866 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T09:37:40.974 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T09:37:40.974 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T09:37:41.226 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:41.246 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librbd1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:41.351 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:41.372 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-09T09:37:41.462 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-09T09:37:41.503 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-09T09:37:41.700 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-09T09:37:41.834 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T09:37:41.954 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:42.176 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T09:37:42.176 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T09:37:42.471 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:42.511 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:42.589 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:42.637 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-09T09:37:42.984 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:43.029 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T09:37:43.029 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T09:37:43.106 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:43.241 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:43.291 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:43.358 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:43.434 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:43.777 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:43.918 INFO:teuthology.orchestra.run.smithi052.stdout:Adding group ceph....done 2021-03-09T09:37:43.992 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T09:37:43.992 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T09:37:44.278 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:44.314 INFO:teuthology.orchestra.run.smithi052.stdout:Adding system user ceph....done 2021-03-09T09:37:44.495 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T09:37:44.496 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T09:37:44.715 INFO:teuthology.orchestra.run.smithi052.stdout:Setting system user ceph properties....done 2021-03-09T09:37:44.788 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:44.878 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-09T09:37:44.889 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:45.007 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:45.092 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-09T09:37:45.108 INFO:teuthology.orchestra.run.smithi194.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-09T09:37:45.317 INFO:teuthology.orchestra.run.smithi194.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-09T09:37:45.371 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:45.635 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T09:37:45.635 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T09:37:45.998 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:46.148 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:46.489 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-09T09:37:46.759 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:46.965 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T09:37:46.965 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T09:37:47.260 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:47.403 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:47.569 INFO:teuthology.orchestra.run.smithi194.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-09T09:37:47.612 INFO:teuthology.orchestra.run.smithi194.stdout:ureadahead will be reprofiled on next reboot 2021-03-09T09:37:47.670 INFO:teuthology.orchestra.run.smithi194.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-09T09:37:47.865 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T09:37:47.866 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T09:37:48.122 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:48.122 INFO:teuthology.orchestra.run.smithi194.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-09T09:37:48.125 DEBUG:teuthology.parallel:result is None 2021-03-09T09:37:48.265 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:48.407 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:48.630 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T09:37:48.630 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T09:37:48.883 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:49.097 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T09:37:49.097 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T09:37:49.385 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:49.511 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:49.830 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:37:49.956 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-09T09:37:50.149 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-09T09:37:52.260 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-09T09:37:52.294 INFO:teuthology.orchestra.run.smithi052.stdout:ureadahead will be reprofiled on next reboot 2021-03-09T09:37:52.352 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-09T09:37:52.797 INFO:teuthology.orchestra.run.smithi052.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-09T09:37:52.799 DEBUG:teuthology.parallel:result is None 2021-03-09T09:37:52.800 DEBUG:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2021-03-09T09:37:52.854 INFO:teuthology.orchestra.run.smithi052.stdout:Ubuntu 2021-03-09T09:37:52.859 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:37:52.860 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T09:37:53.014 DEBUG:teuthology.orchestra.run.smithi052:> dpkg-query -W -f '${Version}' ceph 2021-03-09T09:37:53.038 INFO:teuthology.orchestra.run.smithi052.stdout:17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:37:53.039 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:37:53.039 INFO:teuthology.task.install:The correct ceph version 17.0.0-1742-g89fb622b-1bionic is installed. 2021-03-09T09:37:53.040 DEBUG:teuthology.orchestra.run.smithi194:> sudo lsb_release -is 2021-03-09T09:37:53.096 INFO:teuthology.orchestra.run.smithi194.stdout:Ubuntu 2021-03-09T09:37:53.102 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:37:53.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T09:37:53.254 DEBUG:teuthology.orchestra.run.smithi194:> dpkg-query -W -f '${Version}' ceph 2021-03-09T09:37:53.280 INFO:teuthology.orchestra.run.smithi194.stdout:17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:37:53.281 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:37:53.281 INFO:teuthology.task.install:The correct ceph version 17.0.0-1742-g89fb622b-1bionic is installed. 2021-03-09T09:37:53.282 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-03-09T09:37:53.285 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:37:53.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-09T09:37:53.301 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:37:53.302 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-09T09:37:53.343 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-03-09T09:37:53.344 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:37:53.344 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2021-03-09T09:37:53.362 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-09T09:37:53.418 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:37:53.418 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2021-03-09T09:37:53.434 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-09T09:37:53.488 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-03-09T09:37:53.489 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:37:53.490 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-09T09:37:53.504 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-09T09:37:53.563 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:37:53.564 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-09T09:37:53.578 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-09T09:37:53.633 INFO:teuthology.run_tasks:Running task ceph... 2021-03-09T09:37:53.714 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-03-09T09:37:53.715 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /var/log/ceph 2021-03-09T09:37:53.718 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /var/log/ceph 2021-03-09T09:37:53.732 INFO:tasks.ceph:Disabling ceph logrotate... 2021-03-09T09:37:53.733 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-09T09:37:53.774 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-09T09:37:53.801 INFO:tasks.ceph:Creating extra log directories... 2021-03-09T09:37:53.801 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-09T09:37:53.830 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-09T09:37:53.890 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2021-03-09T09:37:53.892 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2021-03-09T09:37:53.893 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2021-03-09T09:37:53.894 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:37:53.894 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-03-09T09:37:53.901 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-03-09T09:37:53.978 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:37:53.979 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-03-09T09:37:53.986 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-03-09T09:37:54.067 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-03-09T09:37:54.067 INFO:tasks.ceph:config {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug mgrc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_', '\\(RECENT_CRASH\\)', 'replacing it with standby', 'No standby daemons available', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-03-09T09:37:54.068 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949129', 'branch': 'wip-kefu-testing-2021-03-09-1428', 'description': 'rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/crash}', 'email': 'tchaikov@gmail.com', 'first_in_suite': False, 'job_id': '5949129', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 2, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '18.04', 'overrides': {'admin_socket': {'branch': 'wip-kefu-testing-2021-03-09-1428'}, 'ceph': {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug mgrc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-kefu-testing-2021-03-09-1428', 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}}, 'owner': 'scheduled_kchai@teuthology', 'priority': 60, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'], ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']], 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'wip-kefu-testing-2021-03-09-1428', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'targets': {'smithi052.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD0ZQp1ZeElCiOPSkQH7WdORAWptn1TAh5z/J44mMHlsgABBAat/r0NZfbQkgkGygkwfwvY8RMvHvDXG5ldpAeIN7vRK0hoVx1iHY1rNA+E1CEV+vMSK43oXdiARX59vGm0X4Y5qgADJsFHMzzbvzXViev3LXiZwvgFig3PBstn/k4OY7kd97wqaM8bhhwKtf0d+lJw6o+s0rco7oThxj/Cw+7R2OKFRWnStGn2u8d1KtXtVGk16nNXDDLbr3XAyppKCkhIATtGcKKoEQlFDYgRK2Rp2pxKuzeFQ4gXhugxWKKAVh2DiIxNuWJworpwesE/cAZ2w/U5kJJgr1ELHlsv', 'smithi194.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCv/RHVZrPckeonnNJlHjUDhH9eL2nn5+KFKF90W7xHY3W4jTznBIlKDGtb/LfK+Sv0CBi8T0g9mLJ618uiFS6qQFqtCS45ibuedVAhW+thV530gzC4DVWok85idX2rrG7mvafE32HsqioMpoaZ2zvYHzDaMRJ5X6beBNNthT4qL1MuzPwP6giVCp5tyQyae1T9LfykU4ZDZj2wUPBhFMwvip5MO3iAhFoz7kxEVBBx+KVFZ76sQVHZ74wEBaZOcthg6SQQYlbXVQAME2eC4+/JrdWM4YfIwjbGPdbjfTSvY/aEhzpkuvzlviaKyzkzfHxRUyw9DZC/nhDsPe5NlVTj'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_', '\\(RECENT_CRASH\\)', 'replacing it with standby', 'No standby daemons available', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'wait-for-scrub': False, 'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug mgrc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'cluster': 'ceph'}}, {'cephfs_test_runner': {'modules': ['tasks.mgr.test_crash']}}], 'teuthology_branch': 'master', 'teuthology_sha1': '3082387bba74fcd24c9700593d10418152d53c97', 'timestamp': '2021-03-09_08:22:40', 'tube': 'smithi', 'user': 'kchai', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.25973'} 2021-03-09T09:37:54.069 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-03-09T09:37:54.071 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-03-09T09:37:54.109 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2021-03-09T09:37:54.121 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/run/ceph 2021-03-09T09:37:54.164 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:37:54.165 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2021-03-09T09:37:54.182 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-09T09:37:54.182 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-09T09:37:54.230 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-09T09:37:54.231 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:37:54.231 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 340 Links: 1 2021-03-09T09:37:54.231 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:37:54.231 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-03-09 09:35:22.513605739 +0000 2021-03-09T09:37:54.232 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-03-09 09:35:22.385605741 +0000 2021-03-09T09:37:54.232 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-03-09 09:35:22.385605741 +0000 2021-03-09T09:37:54.232 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-03-09T09:37:54.240 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-03-09T09:37:54.240 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-03-09T09:37:54.241 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000305667 s, 1.7 MB/s 2021-03-09T09:37:54.245 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-09T09:37:54.294 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-09T09:37:54.295 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:37:54.295 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 347 Links: 1 2021-03-09T09:37:54.295 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:37:54.295 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-03-09 09:35:22.757605737 +0000 2021-03-09T09:37:54.296 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-03-09 09:35:22.661605738 +0000 2021-03-09T09:37:54.296 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-03-09 09:35:22.661605738 +0000 2021-03-09T09:37:54.296 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-03-09T09:37:54.304 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-03-09T09:37:54.304 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-03-09T09:37:54.305 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000295706 s, 1.7 MB/s 2021-03-09T09:37:54.309 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-09T09:37:54.358 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-09T09:37:54.359 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:37:54.359 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 353 Links: 1 2021-03-09T09:37:54.359 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:37:54.360 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-03-09 09:35:22.997605734 +0000 2021-03-09T09:37:54.360 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-03-09 09:35:22.893605735 +0000 2021-03-09T09:37:54.360 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-03-09 09:35:22.893605735 +0000 2021-03-09T09:37:54.360 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-03-09T09:37:54.368 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-03-09T09:37:54.369 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-03-09T09:37:54.369 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000306493 s, 1.7 MB/s 2021-03-09T09:37:54.373 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-09T09:37:54.423 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-09T09:37:54.424 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:37:54.424 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 359 Links: 1 2021-03-09T09:37:54.424 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:37:54.424 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-03-09 09:35:23.237605731 +0000 2021-03-09T09:37:54.424 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-03-09 09:35:23.121605733 +0000 2021-03-09T09:37:54.425 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-03-09 09:35:23.121605733 +0000 2021-03-09T09:37:54.425 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-03-09T09:37:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-03-09T09:37:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-03-09T09:37:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000234625 s, 2.2 MB/s 2021-03-09T09:37:54.436 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1'} 2021-03-09T09:37:54.437 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:37:54.437 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2021-03-09T09:37:54.444 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-09T09:37:54.444 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-09T09:37:54.494 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-09T09:37:54.494 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:37:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 340 Links: 1 2021-03-09T09:37:54.495 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:37:54.495 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2021-03-09 09:35:22.455648857 +0000 2021-03-09T09:37:54.495 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2021-03-09 09:35:22.335648860 +0000 2021-03-09T09:37:54.495 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2021-03-09 09:35:22.335648860 +0000 2021-03-09T09:37:54.496 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2021-03-09T09:37:54.503 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2021-03-09T09:37:54.503 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2021-03-09T09:37:54.503 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000311388 s, 1.6 MB/s 2021-03-09T09:37:54.508 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-09T09:37:54.557 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-09T09:37:54.558 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:37:54.558 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 346 Links: 1 2021-03-09T09:37:54.558 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:37:54.559 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2021-03-09 09:35:22.699648852 +0000 2021-03-09T09:37:54.559 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2021-03-09 09:35:22.579648854 +0000 2021-03-09T09:37:54.559 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2021-03-09 09:35:22.579648854 +0000 2021-03-09T09:37:54.559 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2021-03-09T09:37:54.567 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2021-03-09T09:37:54.567 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2021-03-09T09:37:54.567 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000309039 s, 1.7 MB/s 2021-03-09T09:37:54.571 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-09T09:37:54.621 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-09T09:37:54.622 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:37:54.622 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 352 Links: 1 2021-03-09T09:37:54.622 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:37:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2021-03-09 09:35:22.827648849 +0000 2021-03-09T09:37:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2021-03-09 09:35:22.823648849 +0000 2021-03-09T09:37:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2021-03-09 09:35:22.823648849 +0000 2021-03-09T09:37:54.623 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2021-03-09T09:37:54.631 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2021-03-09T09:37:54.631 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2021-03-09T09:37:54.632 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.00041105 s, 1.2 MB/s 2021-03-09T09:37:54.636 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-09T09:37:54.685 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-09T09:37:54.686 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:37:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 359 Links: 1 2021-03-09T09:37:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:37:54.687 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2021-03-09 09:35:23.075648843 +0000 2021-03-09T09:37:54.687 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2021-03-09 09:35:23.071648843 +0000 2021-03-09T09:37:54.687 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2021-03-09 09:35:23.071648843 +0000 2021-03-09T09:37:54.687 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2021-03-09T09:37:54.695 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2021-03-09T09:37:54.695 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2021-03-09T09:37:54.696 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000443904 s, 1.2 MB/s 2021-03-09T09:37:54.700 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2021-03-09T09:37:54.700 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2021-03-09T09:37:54.701 INFO:tasks.ceph:Generating config... 2021-03-09T09:37:54.703 INFO:tasks.ceph:[client] debug client = 20 2021-03-09T09:37:54.703 INFO:tasks.ceph:[client] debug mgrc = 20 2021-03-09T09:37:54.703 INFO:tasks.ceph:[client] debug ms = 1 2021-03-09T09:37:54.704 INFO:tasks.ceph:[global] mon election default strategy = 1 2021-03-09T09:37:54.704 INFO:tasks.ceph:[mds] debug mgrc = 20 2021-03-09T09:37:54.704 INFO:tasks.ceph:[mgr] debug client = 20 2021-03-09T09:37:54.705 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-03-09T09:37:54.705 INFO:tasks.ceph:[mgr] debug ms = 1 2021-03-09T09:37:54.705 INFO:tasks.ceph:[mon] debug mon = 20 2021-03-09T09:37:54.706 INFO:tasks.ceph:[mon] debug ms = 1 2021-03-09T09:37:54.706 INFO:tasks.ceph:[mon] debug paxos = 20 2021-03-09T09:37:54.706 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-03-09T09:37:54.706 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2021-03-09T09:37:54.707 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2021-03-09T09:37:54.707 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-03-09T09:37:54.707 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-03-09T09:37:54.708 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-03-09T09:37:54.708 INFO:tasks.ceph:[osd] debug filestore = 20 2021-03-09T09:37:54.708 INFO:tasks.ceph:[osd] debug journal = 20 2021-03-09T09:37:54.708 INFO:tasks.ceph:[osd] debug mgrc = 20 2021-03-09T09:37:54.709 INFO:tasks.ceph:[osd] debug ms = 20 2021-03-09T09:37:54.709 INFO:tasks.ceph:[osd] debug osd = 25 2021-03-09T09:37:54.709 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-03-09T09:37:54.710 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-03-09T09:37:54.710 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-03-09T09:37:54.710 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-03-09T09:37:54.710 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-03-09T09:37:54.711 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-03-09T09:37:54.711 INFO:tasks.ceph:Setting up mon.a... 2021-03-09T09:37:54.712 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-03-09T09:37:54.800 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/ceph.keyring 2021-03-09T09:37:54.806 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-03-09T09:37:54.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T09:37:54.907 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.52'), ('mon.c', '[v2:172.21.15.52:3301,v1:172.21.15.52:6790]'), ('mon.b', '172.21.15.194')] 2021-03-09T09:37:54.908 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.52,[v2:172.21.15.52:3301,v1:172.21.15.52:6790],172.21.15.194', 'mon election default strategy': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug mgrc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds': {'debug mgrc': 20}} 2021-03-09T09:37:54.909 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:37:54.909 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-03-09T09:37:54.954 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.52 --addv c '[v2:172.21.15.52:3301,v1:172.21.15.52:6790]' --add b 172.21.15.194 --print /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:37:55.088 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:37:55.089 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: generated fsid 1325f083-8043-46e9-93ad-0afef33c673d 2021-03-09T09:37:55.089 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = octopus 2021-03-09T09:37:55.089 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 0 2021-03-09T09:37:55.089 INFO:teuthology.orchestra.run.smithi052.stdout:fsid 1325f083-8043-46e9-93ad-0afef33c673d 2021-03-09T09:37:55.090 INFO:teuthology.orchestra.run.smithi052.stdout:last_changed 2021-03-09T09:37:55.089609+0000 2021-03-09T09:37:55.090 INFO:teuthology.orchestra.run.smithi052.stdout:created 2021-03-09T09:37:55.089609+0000 2021-03-09T09:37:55.090 INFO:teuthology.orchestra.run.smithi052.stdout:min_mon_release 15 (octopus) 2021-03-09T09:37:55.090 INFO:teuthology.orchestra.run.smithi052.stdout:election_strategy: 1 2021-03-09T09:37:55.090 INFO:teuthology.orchestra.run.smithi052.stdout:0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2021-03-09T09:37:55.091 INFO:teuthology.orchestra.run.smithi052.stdout:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2021-03-09T09:37:55.091 INFO:teuthology.orchestra.run.smithi052.stdout:2: [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] mon.c 2021-03-09T09:37:55.091 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-03-09T09:37:55.094 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-03-09T09:37:55.101 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1325f083-8043-46e9-93ad-0afef33c673d... 2021-03-09T09:37:55.102 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-03-09T09:37:55.150 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-03-09T09:37:55.179 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2021-03-09T09:37:55.179 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2021-03-09T09:37:55.180 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-09T09:37:55.180 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2021-03-09T09:37:55.180 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.180 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2021-03-09T09:37:55.181 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.181 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2021-03-09T09:37:55.181 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.181 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2021-03-09T09:37:55.181 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2021-03-09T09:37:55.182 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.182 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2021-03-09T09:37:55.182 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2021-03-09T09:37:55.182 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.183 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-09T09:37:55.183 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2021-03-09T09:37:55.183 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.183 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2021-03-09T09:37:55.183 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-09T09:37:55.184 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.184 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2021-03-09T09:37:55.184 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.184 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2021-03-09T09:37:55.185 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2021-03-09T09:37:55.185 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2021-03-09T09:37:55.185 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2021-03-09T09:37:55.185 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2021-03-09T09:37:55.185 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2021-03-09T09:37:55.186 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2021-03-09T09:37:55.186 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-09T09:37:55.186 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-09T09:37:55.186 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2021-03-09T09:37:55.187 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.187 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-09T09:37:55.187 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.188 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2021-03-09T09:37:55.188 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.188 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2021-03-09T09:37:55.188 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.188 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2021-03-09T09:37:55.189 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2021-03-09T09:37:55.189 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2021-03-09T09:37:55.189 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = "172.21.15.52,[v2:172.21.15.52:3301,v1:172.21.15.52:6790],172.21.15.194" 2021-03-09T09:37:55.189 INFO:teuthology.orchestra.run.smithi052.stdout: mon election default strategy = 1 2021-03-09T09:37:55.190 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 1325f083-8043-46e9-93ad-0afef33c673d 2021-03-09T09:37:55.190 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.190 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2021-03-09T09:37:55.190 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2021-03-09T09:37:55.191 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.191 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2021-03-09T09:37:55.191 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2021-03-09T09:37:55.191 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.191 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2021-03-09T09:37:55.192 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2021-03-09T09:37:55.192 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.192 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2021-03-09T09:37:55.192 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2021-03-09T09:37:55.193 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2021-03-09T09:37:55.193 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.193 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2021-03-09T09:37:55.193 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2021-03-09T09:37:55.193 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.194 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2021-03-09T09:37:55.194 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.194 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2021-03-09T09:37:55.194 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.195 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2021-03-09T09:37:55.195 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.195 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2021-03-09T09:37:55.195 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2021-03-09T09:37:55.195 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.196 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2021-03-09T09:37:55.196 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2021-03-09T09:37:55.196 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore block size = 96636764160 2021-03-09T09:37:55.197 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore compression algorithm = snappy 2021-03-09T09:37:55.197 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore compression mode = aggressive 2021-03-09T09:37:55.197 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore fsck on mount = True 2021-03-09T09:37:55.197 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluefs = 20 2021-03-09T09:37:55.197 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluestore = 20 2021-03-09T09:37:55.198 INFO:teuthology.orchestra.run.smithi052.stdout: debug filestore = 20 2021-03-09T09:37:55.198 INFO:teuthology.orchestra.run.smithi052.stdout: debug journal = 20 2021-03-09T09:37:55.198 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgrc = 20 2021-03-09T09:37:55.198 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 20 2021-03-09T09:37:55.198 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 25 2021-03-09T09:37:55.199 INFO:teuthology.orchestra.run.smithi052.stdout: debug rocksdb = 10 2021-03-09T09:37:55.199 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd backfillfull_ratio = 0.85 2021-03-09T09:37:55.199 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd full ratio = 0.9 2021-03-09T09:37:55.199 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd nearfull ratio = 0.8 2021-03-09T09:37:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: osd failsafe full ratio = 0.95 2021-03-09T09:37:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: osd objectstore = bluestore 2021-03-09T09:37:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.200 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2021-03-09T09:37:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-03-09T09:37:55.201 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2021-03-09T09:37:55.201 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2021-03-09T09:37:55.201 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2021-03-09T09:37:55.201 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2021-03-09T09:37:55.201 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2021-03-09T09:37:55.202 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2021-03-09T09:37:55.202 INFO:teuthology.orchestra.run.smithi052.stdout: debug client = 20 2021-03-09T09:37:55.202 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.202 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2021-03-09T09:37:55.202 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-03-09T09:37:55.203 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2021-03-09T09:37:55.203 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2021-03-09T09:37:55.203 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2021-03-09T09:37:55.203 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2021-03-09T09:37:55.203 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2021-03-09T09:37:55.204 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2021-03-09T09:37:55.204 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2021-03-09T09:37:55.204 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2021-03-09T09:37:55.204 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2021-03-09T09:37:55.204 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-09T09:37:55.205 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2021-03-09T09:37:55.205 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2021-03-09T09:37:55.205 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2021-03-09T09:37:55.205 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2021-03-09T09:37:55.206 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-09T09:37:55.206 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-09T09:37:55.206 INFO:teuthology.orchestra.run.smithi052.stdout: debug client = 20 2021-03-09T09:37:55.206 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgrc = 20 2021-03-09T09:37:55.206 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-03-09T09:37:55.207 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2021-03-09T09:37:55.207 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.c] 2021-03-09T09:37:55.207 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.b] 2021-03-09T09:37:55.207 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.a] 2021-03-09T09:37:55.207 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.c] 2021-03-09T09:37:55.208 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.b] 2021-03-09T09:37:55.208 INFO:teuthology.orchestra.run.smithi052.stdout:[mds] 2021-03-09T09:37:55.208 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgrc = 20 2021-03-09T09:37:55.210 INFO:teuthology.orchestra.run.smithi194.stdout:[global] 2021-03-09T09:37:55.210 INFO:teuthology.orchestra.run.smithi194.stdout: chdir = "" 2021-03-09T09:37:55.211 INFO:teuthology.orchestra.run.smithi194.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-09T09:37:55.211 INFO:teuthology.orchestra.run.smithi194.stdout: auth supported = cephx 2021-03-09T09:37:55.211 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.211 INFO:teuthology.orchestra.run.smithi194.stdout: filestore xattr use omap = true 2021-03-09T09:37:55.212 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.212 INFO:teuthology.orchestra.run.smithi194.stdout: mon clock drift allowed = 1.000 2021-03-09T09:37:55.212 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.212 INFO:teuthology.orchestra.run.smithi194.stdout: osd crush chooseleaf type = 0 2021-03-09T09:37:55.212 INFO:teuthology.orchestra.run.smithi194.stdout: auth debug = true 2021-03-09T09:37:55.213 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.213 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on old message = true 2021-03-09T09:37:55.213 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on bug = true 2021-03-09T09:37:55.213 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.213 INFO:teuthology.orchestra.run.smithi194.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-09T09:37:55.214 INFO:teuthology.orchestra.run.smithi194.stdout: mon pg warn max object skew = 0 2021-03-09T09:37:55.214 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.214 INFO:teuthology.orchestra.run.smithi194.stdout: # disable pg_autoscaler by default for new pools 2021-03-09T09:37:55.214 INFO:teuthology.orchestra.run.smithi194.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-09T09:37:55.215 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.215 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default size = 2 2021-03-09T09:37:55.215 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.215 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow primary affinity = true 2021-03-09T09:37:55.215 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow pg remap = true 2021-03-09T09:37:55.216 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on legacy crush tunables = false 2021-03-09T09:37:55.216 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on crush straw calc version zero = false 2021-03-09T09:37:55.216 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on no sortbitwise = false 2021-03-09T09:37:55.216 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on osd down out interval zero = false 2021-03-09T09:37:55.217 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on too few osds = false 2021-03-09T09:37:55.217 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-09T09:37:55.217 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-09T09:37:55.217 INFO:teuthology.orchestra.run.smithi194.stdout: mon_allow_pool_size_one = true 2021-03-09T09:37:55.218 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.218 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-09T09:37:55.218 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.218 INFO:teuthology.orchestra.run.smithi194.stdout: osd default data pool replay window = 5 2021-03-09T09:37:55.218 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.219 INFO:teuthology.orchestra.run.smithi194.stdout: mon allow pool delete = true 2021-03-09T09:37:55.219 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.219 INFO:teuthology.orchestra.run.smithi194.stdout: mon cluster log file level = debug 2021-03-09T09:37:55.219 INFO:teuthology.orchestra.run.smithi194.stdout: debug asserts on shutdown = true 2021-03-09T09:37:55.219 INFO:teuthology.orchestra.run.smithi194.stdout: mon health detail to clog = false 2021-03-09T09:37:55.220 INFO:teuthology.orchestra.run.smithi194.stdout: mon host = "172.21.15.52,[v2:172.21.15.52:3301,v1:172.21.15.52:6790],172.21.15.194" 2021-03-09T09:37:55.220 INFO:teuthology.orchestra.run.smithi194.stdout: mon election default strategy = 1 2021-03-09T09:37:55.220 INFO:teuthology.orchestra.run.smithi194.stdout: fsid = 1325f083-8043-46e9-93ad-0afef33c673d 2021-03-09T09:37:55.220 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.221 INFO:teuthology.orchestra.run.smithi194.stdout:[osd] 2021-03-09T09:37:55.221 INFO:teuthology.orchestra.run.smithi194.stdout: osd journal size = 100 2021-03-09T09:37:55.221 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.221 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub load threshold = 5.0 2021-03-09T09:37:55.222 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub max interval = 600 2021-03-09T09:37:55.222 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.222 INFO:teuthology.orchestra.run.smithi194.stdout: osd recover clone overlap = true 2021-03-09T09:37:55.222 INFO:teuthology.orchestra.run.smithi194.stdout: osd recovery max chunk = 1048576 2021-03-09T09:37:55.222 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.223 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug shutdown = true 2021-03-09T09:37:55.223 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug op order = true 2021-03-09T09:37:55.223 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify stray on activate = true 2021-03-09T09:37:55.223 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.223 INFO:teuthology.orchestra.run.smithi194.stdout: osd open classes on start = true 2021-03-09T09:37:55.224 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug pg log writeout = true 2021-03-09T09:37:55.224 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.224 INFO:teuthology.orchestra.run.smithi194.stdout: osd deep scrub update digest min age = 30 2021-03-09T09:37:55.224 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.225 INFO:teuthology.orchestra.run.smithi194.stdout: osd map max advance = 10 2021-03-09T09:37:55.225 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.225 INFO:teuthology.orchestra.run.smithi194.stdout: journal zero on create = true 2021-03-09T09:37:55.225 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.225 INFO:teuthology.orchestra.run.smithi194.stdout: filestore ondisk finisher threads = 3 2021-03-09T09:37:55.226 INFO:teuthology.orchestra.run.smithi194.stdout: filestore apply finisher threads = 3 2021-03-09T09:37:55.226 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.226 INFO:teuthology.orchestra.run.smithi194.stdout: bdev debug aio = true 2021-03-09T09:37:55.226 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug misdirected ops = true 2021-03-09T09:37:55.227 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore block size = 96636764160 2021-03-09T09:37:55.227 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore compression algorithm = snappy 2021-03-09T09:37:55.227 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore compression mode = aggressive 2021-03-09T09:37:55.227 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore fsck on mount = True 2021-03-09T09:37:55.227 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluefs = 20 2021-03-09T09:37:55.228 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluestore = 20 2021-03-09T09:37:55.228 INFO:teuthology.orchestra.run.smithi194.stdout: debug filestore = 20 2021-03-09T09:37:55.228 INFO:teuthology.orchestra.run.smithi194.stdout: debug journal = 20 2021-03-09T09:37:55.228 INFO:teuthology.orchestra.run.smithi194.stdout: debug mgrc = 20 2021-03-09T09:37:55.229 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 20 2021-03-09T09:37:55.229 INFO:teuthology.orchestra.run.smithi194.stdout: debug osd = 25 2021-03-09T09:37:55.229 INFO:teuthology.orchestra.run.smithi194.stdout: debug rocksdb = 10 2021-03-09T09:37:55.229 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd backfillfull_ratio = 0.85 2021-03-09T09:37:55.229 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd full ratio = 0.9 2021-03-09T09:37:55.230 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd nearfull ratio = 0.8 2021-03-09T09:37:55.230 INFO:teuthology.orchestra.run.smithi194.stdout: osd failsafe full ratio = 0.95 2021-03-09T09:37:55.230 INFO:teuthology.orchestra.run.smithi194.stdout: osd objectstore = bluestore 2021-03-09T09:37:55.230 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.230 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr] 2021-03-09T09:37:55.231 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2021-03-09T09:37:55.231 INFO:teuthology.orchestra.run.smithi194.stdout: debug mgr = 20 2021-03-09T09:37:55.231 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2021-03-09T09:37:55.231 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2021-03-09T09:37:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2021-03-09T09:37:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2021-03-09T09:37:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: mgr/telemetry/nag = false 2021-03-09T09:37:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: debug client = 20 2021-03-09T09:37:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.233 INFO:teuthology.orchestra.run.smithi194.stdout:[mon] 2021-03-09T09:37:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2021-03-09T09:37:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2021-03-09T09:37:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: debug paxos = 20 2021-03-09T09:37:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2021-03-09T09:37:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: mon data avail warn = 5 2021-03-09T09:37:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: mon mgr mkfs grace = 240 2021-03-09T09:37:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2021-03-09T09:37:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd reporter subtree level = osd 2021-03-09T09:37:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd prime pg temp = true 2021-03-09T09:37:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2021-03-09T09:37:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-03-09T09:37:55.235 INFO:teuthology.orchestra.run.smithi194.stdout:[client] 2021-03-09T09:37:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: rgw cache enabled = true 2021-03-09T09:37:55.236 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable ops log = true 2021-03-09T09:37:55.236 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable usage log = true 2021-03-09T09:37:55.236 INFO:teuthology.orchestra.run.smithi194.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-09T09:37:55.237 INFO:teuthology.orchestra.run.smithi194.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-09T09:37:55.237 INFO:teuthology.orchestra.run.smithi194.stdout: debug client = 20 2021-03-09T09:37:55.237 INFO:teuthology.orchestra.run.smithi194.stdout: debug mgrc = 20 2021-03-09T09:37:55.237 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2021-03-09T09:37:55.237 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.a] 2021-03-09T09:37:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.c] 2021-03-09T09:37:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.b] 2021-03-09T09:37:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:[mds.a] 2021-03-09T09:37:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:[mds.c] 2021-03-09T09:37:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:[mds.b] 2021-03-09T09:37:55.239 INFO:teuthology.orchestra.run.smithi194.stdout:[mds] 2021-03-09T09:37:55.239 INFO:teuthology.orchestra.run.smithi194.stdout: debug mgrc = 20 2021-03-09T09:37:55.239 INFO:tasks.ceph:Creating admin key on mon.a... 2021-03-09T09:37:55.240 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-03-09T09:37:55.359 INFO:tasks.ceph:Copying monmap to all nodes... 2021-03-09T09:37:55.359 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:37:55.360 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2021-03-09T09:37:55.364 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:37:55.365 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2021-03-09T09:37:55.412 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2021-03-09T09:37:55.413 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:37:55.414 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.keyring 2021-03-09T09:37:55.414 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T09:37:55.475 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:37:55.476 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:37:55.521 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi194.front.sepia.ceph.com 2021-03-09T09:37:55.522 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:37:55.522 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.keyring 2021-03-09T09:37:55.523 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T09:37:55.548 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:37:55.548 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:37:55.597 INFO:tasks.ceph:Setting up mon nodes... 2021-03-09T09:37:55.597 INFO:tasks.ceph:Setting up mgr nodes... 2021-03-09T09:37:55.598 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-03-09T09:37:55.691 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-03-09T09:37:55.697 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2021-03-09T09:37:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2021-03-09T09:37:55.803 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mgr/ceph-z && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.z /var/lib/ceph/mgr/ceph-z/keyring 2021-03-09T09:37:55.891 INFO:teuthology.orchestra.run.smithi194.stdout:creating /var/lib/ceph/mgr/ceph-z/keyring 2021-03-09T09:37:55.897 INFO:tasks.ceph:Setting up mds nodes... 2021-03-09T09:37:55.897 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2021-03-09T09:37:55.998 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2021-03-09T09:37:56.005 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2021-03-09T09:37:56.020 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2021-03-09T09:37:56.162 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2021-03-09T09:37:56.169 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2021-03-09T09:37:56.180 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2021-03-09T09:37:56.273 INFO:teuthology.orchestra.run.smithi194.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2021-03-09T09:37:56.279 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2021-03-09T09:37:56.289 INFO:tasks.ceph_client:Setting up client nodes... 2021-03-09T09:37:56.290 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-03-09T09:37:56.381 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/ceph.client.0.keyring 2021-03-09T09:37:56.397 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-03-09T09:37:56.485 INFO:teuthology.orchestra.run.smithi194.stdout:creating /etc/ceph/ceph.client.1.keyring 2021-03-09T09:37:56.497 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-03-09T09:37:56.498 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2021-03-09T09:37:56.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-03-09T09:37:56.514 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1'} 2021-03-09T09:37:56.514 INFO:tasks.ceph:role: osd.0 2021-03-09T09:37:56.515 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com 2021-03-09T09:37:56.515 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-09T09:37:56.601 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T09:37:56.602 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T09:37:56.602 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T09:37:56.602 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T09:37:56.602 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2021-03-09T09:37:56.603 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T09:37:56.603 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T09:37:56.603 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T09:37:56.603 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T09:37:56.604 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2021-03-09T09:37:56.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-03-09T09:37:56.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-03-09T09:37:56.723 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T09:37:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T09:37:56.725 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:37:56.894 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:37:56.891+0000 7fed7600df00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-03-09T09:37:56.895 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:37:56.891+0000 7fed7600df00 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-03-09T09:37:56.895 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:37:56.891+0000 7fed7600df00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T09:37:56.895 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:37:56.891+0000 7fed7600df00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T09:37:56.895 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:37:56.891+0000 7fed7600df00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T09:37:56.896 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:37:56.891+0000 7fed7600df00 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-03-09T09:37:59.897 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-03-09T09:37:59.923 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-03-09T09:37:59.935 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2021-03-09T09:37:59.936 INFO:tasks.ceph:role: osd.1 2021-03-09T09:37:59.936 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com 2021-03-09T09:37:59.936 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-09T09:38:00.470 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T09:38:00.470 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T09:38:00.471 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T09:38:00.471 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T09:38:00.471 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2021-03-09T09:38:00.472 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T09:38:00.472 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T09:38:00.473 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T09:38:00.473 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T09:38:00.475 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2021-03-09T09:38:00.476 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2021-03-09T09:38:00.539 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-03-09T09:38:00.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T09:38:00.548 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T09:38:00.548 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-03-09T09:38:00.605 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2021-03-09T09:38:00.605 INFO:tasks.ceph:role: osd.2 2021-03-09T09:38:00.606 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com 2021-03-09T09:38:00.606 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-09T09:38:01.129 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T09:38:01.129 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T09:38:01.129 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T09:38:01.130 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T09:38:01.130 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2021-03-09T09:38:01.130 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T09:38:01.130 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T09:38:01.131 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T09:38:01.131 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T09:38:01.131 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2021-03-09T09:38:01.132 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2021-03-09T09:38:01.205 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-03-09T09:38:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T09:38:01.217 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T09:38:01.217 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-03-09T09:38:01.277 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2021-03-09T09:38:01.278 INFO:tasks.ceph:role: osd.3 2021-03-09T09:38:01.278 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com 2021-03-09T09:38:01.279 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-09T09:38:01.774 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T09:38:01.775 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T09:38:01.775 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T09:38:01.775 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T09:38:01.775 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2021-03-09T09:38:01.776 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T09:38:01.776 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T09:38:01.776 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T09:38:01.776 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T09:38:01.777 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2021-03-09T09:38:01.778 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2021-03-09T09:38:01.827 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-03-09T09:38:01.886 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T09:38:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T09:38:01.888 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:38:02.053 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:02.051+0000 7f062a241f00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-03-09T09:38:02.056 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:02.055+0000 7f062a241f00 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-03-09T09:38:02.056 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:02.055+0000 7f062a241f00 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T09:38:02.056 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:02.055+0000 7f062a241f00 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T09:38:02.057 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:02.055+0000 7f062a241f00 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T09:38:02.059 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:02.059+0000 7f062a241f00 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-03-09T09:38:05.021 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-03-09T09:38:05.036 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:38:05.174 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:05.171+0000 7f202086df00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-03-09T09:38:05.174 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:05.171+0000 7f202086df00 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-03-09T09:38:05.174 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:05.171+0000 7f202086df00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T09:38:05.175 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:05.171+0000 7f202086df00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T09:38:05.175 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:05.171+0000 7f202086df00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T09:38:05.175 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:05.171+0000 7f202086df00 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-03-09T09:38:08.153 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-03-09T09:38:08.176 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:38:08.321 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:08.315+0000 7f48d6827f00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-03-09T09:38:08.322 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:08.315+0000 7f48d6827f00 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-03-09T09:38:08.322 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:08.315+0000 7f48d6827f00 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T09:38:08.323 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:08.315+0000 7f48d6827f00 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T09:38:08.323 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:08.315+0000 7f48d6827f00 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T09:38:08.324 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:08.319+0000 7f48d6827f00 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-03-09T09:38:11.300 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-03-09T09:38:11.311 INFO:tasks.ceph:Reading keys from all nodes... 2021-03-09T09:38:11.312 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:38:11.313 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2021-03-09T09:38:11.329 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:38:11.329 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2021-03-09T09:38:11.383 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:38:11.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2021-03-09T09:38:11.435 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:38:11.435 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2021-03-09T09:38:11.494 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:38:11.494 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2021-03-09T09:38:11.509 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:38:11.510 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/mgr/ceph-z/keyring of=/dev/stdout 2021-03-09T09:38:11.565 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:38:11.566 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2021-03-09T09:38:11.620 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:38:11.620 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2021-03-09T09:38:11.676 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:38:11.677 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2021-03-09T09:38:11.741 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:38:11.742 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2021-03-09T09:38:11.797 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:38:11.797 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2021-03-09T09:38:11.804 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:38:11.804 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2021-03-09T09:38:11.843 INFO:tasks.ceph:Adding keys to all mons... 2021-03-09T09:38:11.844 DEBUG:teuthology.orchestra.run.smithi052:> sudo tee -a /etc/ceph/ceph.keyring 2021-03-09T09:38:11.849 DEBUG:teuthology.orchestra.run.smithi194:> sudo tee -a /etc/ceph/ceph.keyring 2021-03-09T09:38:11.889 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.x] 2021-03-09T09:38:11.889 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDzQUdg0QhOKRAAWnvBFhsg/EI7Bzl+pFb8Dg== 2021-03-09T09:38:11.889 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.a] 2021-03-09T09:38:11.889 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDzQUdgGdOZOxAAYHwwZ3LPSdouRG7QrKVEkw== 2021-03-09T09:38:11.890 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.c] 2021-03-09T09:38:11.890 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD0QUdgw7bGCRAA/ceXB2+b6w8BodGD/5/eCw== 2021-03-09T09:38:11.890 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2021-03-09T09:38:11.891 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD0QUdgH6FcNRAAJczQm9hU0APbIJyKLeiPOA== 2021-03-09T09:38:11.891 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.y] 2021-03-09T09:38:11.891 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDzQUdg03ecLxAAA5p0weouK/WaNTUeaUD64Q== 2021-03-09T09:38:11.891 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.z] 2021-03-09T09:38:11.891 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDzQUdgj3o2NRAATCDHMWINnCR+Ah5fOFn5nQ== 2021-03-09T09:38:11.892 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.b] 2021-03-09T09:38:11.892 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD0QUdgHgJhEBAAXxygiZALXJp3pkO8631VzQ== 2021-03-09T09:38:11.892 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2021-03-09T09:38:11.892 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD6QUdglE1MAxAAwAx0tl3a1NyuynZkzyPkMQ== 2021-03-09T09:38:11.893 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2021-03-09T09:38:11.893 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD9QUdgZmxvChAAI/hRwzha+eENFVOEW0xQpw== 2021-03-09T09:38:11.893 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.3] 2021-03-09T09:38:11.893 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQAAQkdg+2gUExAAKkZFwmZ0XBgIiz2L/0iboA== 2021-03-09T09:38:11.893 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2021-03-09T09:38:11.894 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD0QUdgUYPRFhAAUWh0DLOtxU0wEBpVFbTxBg== 2021-03-09T09:38:11.894 INFO:teuthology.orchestra.run.smithi052.stdout:[client.1] 2021-03-09T09:38:11.894 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD0QUdglpYEHRAA+ckDbT0mjDQM/5HFwxa1nw== 2021-03-09T09:38:11.899 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.x] 2021-03-09T09:38:11.899 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDzQUdg0QhOKRAAWnvBFhsg/EI7Bzl+pFb8Dg== 2021-03-09T09:38:11.899 INFO:teuthology.orchestra.run.smithi194.stdout:[mds.a] 2021-03-09T09:38:11.900 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDzQUdgGdOZOxAAYHwwZ3LPSdouRG7QrKVEkw== 2021-03-09T09:38:11.900 INFO:teuthology.orchestra.run.smithi194.stdout:[mds.c] 2021-03-09T09:38:11.901 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD0QUdgw7bGCRAA/ceXB2+b6w8BodGD/5/eCw== 2021-03-09T09:38:11.901 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.0] 2021-03-09T09:38:11.901 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD0QUdgH6FcNRAAJczQm9hU0APbIJyKLeiPOA== 2021-03-09T09:38:11.901 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.y] 2021-03-09T09:38:11.901 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDzQUdg03ecLxAAA5p0weouK/WaNTUeaUD64Q== 2021-03-09T09:38:11.902 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.z] 2021-03-09T09:38:11.902 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDzQUdgj3o2NRAATCDHMWINnCR+Ah5fOFn5nQ== 2021-03-09T09:38:11.902 INFO:teuthology.orchestra.run.smithi194.stdout:[mds.b] 2021-03-09T09:38:11.903 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD0QUdgHgJhEBAAXxygiZALXJp3pkO8631VzQ== 2021-03-09T09:38:11.903 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.1] 2021-03-09T09:38:11.903 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD6QUdglE1MAxAAwAx0tl3a1NyuynZkzyPkMQ== 2021-03-09T09:38:11.903 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.2] 2021-03-09T09:38:11.903 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD9QUdgZmxvChAAI/hRwzha+eENFVOEW0xQpw== 2021-03-09T09:38:11.904 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.3] 2021-03-09T09:38:11.904 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQAAQkdg+2gUExAAKkZFwmZ0XBgIiz2L/0iboA== 2021-03-09T09:38:11.904 INFO:teuthology.orchestra.run.smithi194.stdout:[client.0] 2021-03-09T09:38:11.904 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD0QUdgUYPRFhAAUWh0DLOtxU0wEBpVFbTxBg== 2021-03-09T09:38:11.904 INFO:teuthology.orchestra.run.smithi194.stdout:[client.1] 2021-03-09T09:38:11.905 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD0QUdglpYEHRAA+ckDbT0mjDQM/5HFwxa1nw== 2021-03-09T09:38:11.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T09:38:11.938 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T09:38:12.048 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T09:38:12.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T09:38:12.149 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T09:38:12.152 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T09:38:12.246 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:38:12.248 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:38:12.349 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T09:38:12.352 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T09:38:12.446 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.z --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T09:38:12.448 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.z --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T09:38:12.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T09:38:12.547 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T09:38:12.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:38:12.650 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:38:12.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:38:12.746 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:38:12.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:38:12.841 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:38:12.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T09:38:12.938 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T09:38:13.033 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T09:38:13.035 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T09:38:13.130 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-03-09T09:38:13.130 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-03-09T09:38:13.143 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:38:13.299 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-03-09T09:38:13.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-03-09T09:38:13.367 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:38:13.551 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-03-09T09:38:13.564 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-03-09T09:38:13.579 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:38:13.747 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-03-09T09:38:13.762 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:38:13.765 DEBUG:teuthology.orchestra.run.smithi194:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:38:13.808 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-03-09T09:38:13.809 INFO:tasks.ceph.mon.a:Restarting daemon 2021-03-09T09:38:13.810 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-03-09T09:38:13.813 INFO:tasks.ceph.mon.a:Started 2021-03-09T09:38:13.813 INFO:tasks.ceph.mon.c:Restarting daemon 2021-03-09T09:38:13.814 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-03-09T09:38:13.816 INFO:tasks.ceph.mon.c:Started 2021-03-09T09:38:13.817 INFO:tasks.ceph.mon.b:Restarting daemon 2021-03-09T09:38:13.817 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-03-09T09:38:13.852 INFO:tasks.ceph.mon.b:Started 2021-03-09T09:38:13.853 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-03-09T09:38:13.854 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-03-09T09:38:13.855 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-03-09T09:38:13.857 INFO:tasks.ceph.mgr.x:Started 2021-03-09T09:38:13.858 INFO:tasks.ceph.mgr.y:Restarting daemon 2021-03-09T09:38:13.858 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2021-03-09T09:38:13.861 INFO:tasks.ceph.mgr.y:Started 2021-03-09T09:38:13.861 INFO:tasks.ceph.mgr.z:Restarting daemon 2021-03-09T09:38:13.861 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i z 2021-03-09T09:38:13.864 INFO:tasks.ceph.mgr.z:Started 2021-03-09T09:38:13.864 INFO:tasks.ceph:Setting crush tunables to default 2021-03-09T09:38:13.865 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd crush tunables default 2021-03-09T09:38:14.116 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.111+0000 7f6a3e9c2700 1 Processor -- start 2021-03-09T09:38:14.116 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.111+0000 7f6a3e9c2700 1 -- start start 2021-03-09T09:38:14.118 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a3e9c2700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:14.118 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a3e9c2700 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a38140d20 con 0x7f6a380ae0f0 2021-03-09T09:38:14.118 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a3e9c2700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a38062890 con 0x7f6a38061ad0 2021-03-09T09:38:14.118 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a3e9c2700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3813d570 con 0x7f6a380a14d0 2021-03-09T09:38:14.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a37fff700 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f6a380ae0f0 0x7f6a380613c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:50028/0 (socket says 172.21.15.52:50028) 2021-03-09T09:38:14.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a37fff700 1 -- 172.21.15.52:0/2450673713 learned_addr learned my addr 172.21.15.52:0/2450673713 (peer_addr_for_me v1:172.21.15.52:0/0) 2021-03-09T09:38:14.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:14.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a3cf5f700 1 -- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 msgr2=0x7f6a381416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T09:38:14.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a3cf5f700 1 -- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 msgr2=0x7f6a381416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:14.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:14.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.115+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:14.318 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.319+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:14.319 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.319+0000 7f6a3cf5f700 1 -- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 msgr2=0x7f6a381416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T09:38:14.319 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.319+0000 7f6a3cf5f700 1 -- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 msgr2=0x7f6a381416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:14.319 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.319+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:14.320 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.319+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T09:38:14.725 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.723+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:14.725 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.723+0000 7f6a3cf5f700 1 -- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 msgr2=0x7f6a381416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T09:38:14.725 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.723+0000 7f6a3cf5f700 1 -- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 msgr2=0x7f6a381416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:14.725 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.723+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:14.726 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:14.723+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T09:38:15.529 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:15.527+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:15.529 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:15.527+0000 7f6a3cf5f700 1 -- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 msgr2=0x7f6a381416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T09:38:15.529 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:15.527+0000 7f6a3cf5f700 1 -- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 msgr2=0x7f6a381416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:15.530 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:15.527+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:15.530 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:15.527+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T09:38:17.118 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a35ffb700 1 -- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 msgr2=0x7f6a381416d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T09:38:17.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a35ffb700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a380a14d0 0x7f6a381416d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:17.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a35ffb700 1 -- 172.21.15.52:0/2450673713 >> v1:172.21.15.194:6789/0 conn(0x7f6a38061ad0 legacy=0x7f6a38140fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:17.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a35ffb700 1 -- 172.21.15.52:0/2450673713 >> v1:172.21.15.52:6789/0 conn(0x7f6a380ae0f0 legacy=0x7f6a380613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:17.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a35ffb700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:17.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a35ffb700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:17.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a35ffb700 1 -- 172.21.15.52:0/2450673713 --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c0089d0 con 0x7f6a1c0033d0 2021-03-09T09:38:17.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a35ffb700 1 -- 172.21.15.52:0/2450673713 --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c008b40 con 0x7f6a1c005ee0 2021-03-09T09:38:17.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:17.121 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:17.121 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a37fff700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 msgr2=0x7f6a1c005850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T09:38:17.121 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a37fff700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 msgr2=0x7f6a1c005850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:17.121 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:17.121 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:17.122 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 msgr2=0x7f6a1c0083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T09:38:17.122 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 msgr2=0x7f6a1c0083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:17.122 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:17.122 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.115+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:17.319 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.319+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:17.319 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.319+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:17.320 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.319+0000 7f6a37fff700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 msgr2=0x7f6a1c005850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T09:38:17.320 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.319+0000 7f6a37fff700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 msgr2=0x7f6a1c005850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:17.320 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.319+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:17.320 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.319+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T09:38:17.321 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.319+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 msgr2=0x7f6a1c0083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T09:38:17.321 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.319+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 msgr2=0x7f6a1c0083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:17.321 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.319+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:17.321 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.319+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T09:38:17.725 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.723+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:17.725 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.723+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:17.725 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.723+0000 7f6a37fff700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 msgr2=0x7f6a1c005850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T09:38:17.725 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.723+0000 7f6a37fff700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 msgr2=0x7f6a1c005850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:17.726 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.723+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:17.726 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.723+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 msgr2=0x7f6a1c0083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T09:38:17.726 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.723+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 msgr2=0x7f6a1c0083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:17.726 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.723+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:17.727 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.723+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T09:38:17.727 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:17.723+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T09:38:18.529 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:18.527+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:18.529 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:18.527+0000 7f6a37fff700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 msgr2=0x7f6a1c005850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T09:38:18.530 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:18.527+0000 7f6a37fff700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 msgr2=0x7f6a1c005850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:18.530 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:18.527+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:18.530 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:18.527+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:18.531 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:18.527+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T09:38:18.531 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:18.527+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 msgr2=0x7f6a1c0083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T09:38:18.531 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:18.527+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 msgr2=0x7f6a1c0083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:38:18.531 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:18.527+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:18.532 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:18.527+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T09:38:20.118 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a35ffb700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 msgr2=0x7f6a1c0083a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T09:38:20.118 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a35ffb700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c0083a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.118 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a35ffb700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 msgr2=0x7f6a1c005850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T09:38:20.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a35ffb700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0033d0 0x7f6a1c005850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a3cf5f700 1 -- 172.21.15.52:0/2450673713 reap_dead start 2021-03-09T09:38:20.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a35ffb700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c008320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.119 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a35ffb700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c0037e0 0x7f6a1c008ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a35ffb700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0093e0 0x7f6a1c00b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a35ffb700 1 -- 172.21.15.52:0/2450673713 --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c0089d0 con 0x7f6a1c0093e0 2021-03-09T09:38:20.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a35ffb700 1 -- 172.21.15.52:0/2450673713 --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c008b40 con 0x7f6a1c005ee0 2021-03-09T09:38:20.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a35ffb700 1 -- 172.21.15.52:0/2450673713 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c00be40 con 0x7f6a1c0037e0 2021-03-09T09:38:20.120 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c0037e0 0x7f6a1c008ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.121 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0093e0 0x7f6a1c00b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.121 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c008320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.123 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 msgr2=0x7f6a1c008320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T09:38:20.123 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c008320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.123 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0093e0 msgr2=0x7f6a1c00b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T09:38:20.124 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0093e0 0x7f6a1c00b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.124 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a377fe700 1 -- 172.21.15.52:0/2450673713 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a380675c0 con 0x7f6a1c0037e0 2021-03-09T09:38:20.124 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c0037e0 0x7f6a1c008ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6a380aea80 tx=0x7f6a20009040).ready entity=mon.2 client_cookie=a0dcad846c976659 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:20.124 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a36ffd700 1 -- 172.21.15.52:0/2450673713 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20016760 con 0x7f6a1c0037e0 2021-03-09T09:38:20.125 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a36ffd700 1 -- 172.21.15.52:0/2450673713 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a200168c0 con 0x7f6a1c0037e0 2021-03-09T09:38:20.125 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.115+0000 7f6a36ffd700 1 -- 172.21.15.52:0/2450673713 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20016c40 con 0x7f6a1c0037e0 2021-03-09T09:38:20.125 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c0037e0 msgr2=0x7f6a1c008ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:20.125 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c0037e0 0x7f6a1c008ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6a380aea80 tx=0x7f6a20009040).stop 2021-03-09T09:38:20.125 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2450673713 shutdown_connections 2021-03-09T09:38:20.126 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.52:3300/0 conn(0x7f6a1c0093e0 0x7f6a1c00b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.126 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 --2- 172.21.15.52:0/2450673713 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c0037e0 0x7f6a1c008ef0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.126 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 --2- 172.21.15.52:0/2450673713 >> v2:172.21.15.194:3300/0 conn(0x7f6a1c005ee0 0x7f6a1c008320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.126 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2450673713 >> 172.21.15.52:0/2450673713 conn(0x7f6a38065fe0 msgr2=0x7f6a380663f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:20.127 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2450673713 shutdown_connections 2021-03-09T09:38:20.127 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2450673713 wait complete. 2021-03-09T09:38:20.130 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 Processor -- start 2021-03-09T09:38:20.130 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.119+0000 7f6a3e9c2700 1 -- start start 2021-03-09T09:38:20.130 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a3e9c2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a1c0037e0 0x7f6a380a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.130 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a3e9c2700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c005ee0 0x7f6a38143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.131 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a3e9c2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a1c0093e0 0x7f6a381370f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.131 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a3e9c2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a381377b0 con 0x7f6a1c0037e0 2021-03-09T09:38:20.131 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a3e9c2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a38137a50 con 0x7f6a1c0093e0 2021-03-09T09:38:20.131 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a3e9c2700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3809d070 con 0x7f6a1c005ee0 2021-03-09T09:38:20.132 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a377fe700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c005ee0 0x7f6a38143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.132 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a377fe700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c005ee0 0x7f6a38143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3301/0 says I am v2:172.21.15.52:35024/0 (socket says 172.21.15.52:35024) 2021-03-09T09:38:20.132 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a377fe700 1 -- 172.21.15.52:0/2565394384 learned_addr learned my addr 172.21.15.52:0/2565394384 (peer_addr_for_me v2:172.21.15.52:0/0) 2021-03-09T09:38:20.132 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a1c0037e0 0x7f6a380a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.132 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a1c0093e0 0x7f6a381370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.133 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a377fe700 1 -- 172.21.15.52:0/2565394384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a1c0093e0 msgr2=0x7f6a381370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:20.133 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a1c0093e0 0x7f6a381370f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.133 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a377fe700 1 -- 172.21.15.52:0/2565394384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a1c0037e0 msgr2=0x7f6a380a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:20.133 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a1c0037e0 0x7f6a380a0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.134 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a377fe700 1 -- 172.21.15.52:0/2565394384 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3809ee10 con 0x7f6a1c005ee0 2021-03-09T09:38:20.134 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a3cf5f700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a1c0093e0 0x7f6a381370f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-03-09T09:38:20.134 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a37fff700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a1c0037e0 0x7f6a380a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-03-09T09:38:20.134 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a377fe700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c005ee0 0x7f6a38143fd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6a20003ed0 tx=0x7f6a20009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:20.134 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a2ffff700 1 -- 172.21.15.52:0/2565394384 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20016e90 con 0x7f6a1c005ee0 2021-03-09T09:38:20.135 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a2ffff700 1 -- 172.21.15.52:0/2565394384 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a20024440 con 0x7f6a1c005ee0 2021-03-09T09:38:20.135 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2565394384 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a38144a10 con 0x7f6a1c005ee0 2021-03-09T09:38:20.135 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.123+0000 7f6a2ffff700 1 -- 172.21.15.52:0/2565394384 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20024800 con 0x7f6a1c005ee0 2021-03-09T09:38:20.135 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.127+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2565394384 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3809fd00 con 0x7f6a1c005ee0 2021-03-09T09:38:20.135 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.127+0000 7f6a2ffff700 1 -- 172.21.15.52:0/2565394384 <== mon.2 v2:172.21.15.52:3301/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f6a20013040 con 0x7f6a1c005ee0 2021-03-09T09:38:20.136 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.127+0000 7f6a2ffff700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T09:38:20.136 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.127+0000 7f6a2ffff700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T09:38:20.136 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.127+0000 7f6a2ffff700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T09:38:20.136 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.127+0000 7f6a2ffff700 4 mgrc reconnect No active mgr available yet 2021-03-09T09:38:20.137 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.127+0000 7f6a2ffff700 1 -- 172.21.15.52:0/2565394384 <== mon.2 v2:172.21.15.52:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6a2002d420 con 0x7f6a1c005ee0 2021-03-09T09:38:20.137 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.127+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2565394384 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a38045aa0 con 0x7f6a1c005ee0 2021-03-09T09:38:20.137 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.131+0000 7f6a2ffff700 1 -- 172.21.15.52:0/2565394384 <== mon.2 v2:172.21.15.52:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f6a20021430 con 0x7f6a1c005ee0 2021-03-09T09:38:20.232 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.227+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2565394384 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6a38045aa0 con 0x7f6a1c005ee0 2021-03-09T09:38:20.239 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.235+0000 7f6a2ffff700 1 -- 172.21.15.52:0/2565394384 <== mon.2 v2:172.21.15.52:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6a2003a090 con 0x7f6a1c005ee0 2021-03-09T09:38:20.241 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.239+0000 7f6a3e9c2700 10 mgrc shutdown 2021-03-09T09:38:20.241 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.239+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2565394384 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c005ee0 msgr2=0x7f6a38143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:20.241 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.239+0000 7f6a3e9c2700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c005ee0 0x7f6a38143fd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6a20003ed0 tx=0x7f6a20009040).stop 2021-03-09T09:38:20.241 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.239+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2565394384 shutdown_connections 2021-03-09T09:38:20.242 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.239+0000 7f6a3e9c2700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a1c0093e0 0x7f6a381370f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.242 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.239+0000 7f6a3e9c2700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a1c0037e0 0x7f6a380a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.242 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.239+0000 7f6a3e9c2700 1 --2- 172.21.15.52:0/2565394384 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f6a1c005ee0 0x7f6a38143fd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.242 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.239+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2565394384 >> 172.21.15.52:0/2565394384 conn(0x7f6a38065fe0 msgr2=0x7f6a38140fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:20.242 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.239+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2565394384 shutdown_connections 2021-03-09T09:38:20.243 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-09T09:38:20.239+0000 7f6a3e9c2700 1 -- 172.21.15.52:0/2565394384 wait complete. 2021-03-09T09:38:20.243 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2021-03-09T09:38:20.263 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-03-09T09:38:20.263 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-09T09:38:20.264 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2021-03-09T09:38:20.276 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:38:20.277 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2021-03-09T09:38:20.292 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:38:20.293 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2021-03-09T09:38:20.347 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2021-03-09T09:38:20.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2021-03-09T09:38:20.403 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 76d971ae-5519-47c8-a3d6-668100f48891 0 2021-03-09T09:38:20.698 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.695+0000 7f19008bc700 1 Processor -- start 2021-03-09T09:38:20.698 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.695+0000 7f19008bc700 1 -- start start 2021-03-09T09:38:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.695+0000 7f19008bc700 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f18fc0a9a30 0x7f18fc13fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.695+0000 7f19008bc700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc0a75e0 0x7f18fc13ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.695+0000 7f19008bc700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18fc05ce70 con 0x7f18fc05ca90 2021-03-09T09:38:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.695+0000 7f19008bc700 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18fc097800 con 0x7f18fc0a9a30 2021-03-09T09:38:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.695+0000 7f19008bc700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc097ff0 con 0x7f18fc0a75e0 2021-03-09T09:38:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fa59c700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f18fc05ca90 0x7f18fc0a6ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:60534/0 (socket says 172.21.15.194:60534) 2021-03-09T09:38:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fa59c700 1 -- 172.21.15.194:0/2087197991 learned_addr learned my addr 172.21.15.194:0/2087197991 (peer_addr_for_me v1:172.21.15.194:0/0) 2021-03-09T09:38:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fad9d700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc0a75e0 0x7f18fc13ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f9d9b700 1 --2- 172.21.15.194:0/2087197991 >> v2:172.21.15.52:3300/0 conn(0x7f18fc0a9a30 0x7f18fc13fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380245267 0 0) 0x7f18fc05ce70 con 0x7f18fc05ca90 2021-03-09T09:38:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e0003230 con 0x7f18fc05ca90 2021-03-09T09:38:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3872546202 0 0) 0x7f18e0003230 con 0x7f18fc05ca90 2021-03-09T09:38:20.706 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc0a75e0 msgr2=0x7f18fc13ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T09:38:20.706 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc0a75e0 0x7f18fc13ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.706 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 >> v2:172.21.15.52:3300/0 conn(0x7f18fc0a9a30 msgr2=0x7f18fc13fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T09:38:20.706 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 --2- 172.21.15.194:0/2087197991 >> v2:172.21.15.52:3300/0 conn(0x7f18fc0a9a30 0x7f18fc13fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.707 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc098290 con 0x7f18fc05ca90 2021-03-09T09:38:20.707 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3848865948 0 0) 0x7f18ec0014e0 con 0x7f18fc05ca90 2021-03-09T09:38:20.707 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 >> v1:172.21.15.194:6789/0 conn(0x7f18fc05ca90 legacy=0x7f18fc0a6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:20.707 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc0a75e0 0x7f18e0003720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.707 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18e0003c30 0x7f18e00044d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.708 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18e00049c0 0x7f18e0004e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.708 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc097800 con 0x7f18e0003c30 2021-03-09T09:38:20.708 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc097ff0 con 0x7f18e00049c0 2021-03-09T09:38:20.708 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00041f0 con 0x7f18fc0a75e0 2021-03-09T09:38:20.709 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fad9d700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18e00049c0 0x7f18e0004e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.709 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f9d9b700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18e0003c30 0x7f18e00044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.709 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fa59c700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc0a75e0 0x7f18e0003720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.709 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fad9d700 1 -- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc0a75e0 msgr2=0x7f18e0003720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T09:38:20.709 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fad9d700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc0a75e0 0x7f18e0003720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.710 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fad9d700 1 -- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18e0003c30 msgr2=0x7f18e00044d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T09:38:20.710 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fad9d700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18e0003c30 0x7f18e00044d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.710 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fad9d700 1 -- 172.21.15.194:0/2087197991 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0003230 con 0x7f18e00049c0 2021-03-09T09:38:20.710 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fad9d700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18e00049c0 0x7f18e0004e40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f18f4009e60 tx=0x7f18f40091d0).ready entity=mon.1 client_cookie=3daa8e7e9a38f044 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:20.711 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f400edc0 con 0x7f18e00049c0 2021-03-09T09:38:20.711 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f40123f0 con 0x7f18e00049c0 2021-03-09T09:38:20.711 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18f959a700 1 -- 172.21.15.194:0/2087197991 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4012770 con 0x7f18e00049c0 2021-03-09T09:38:20.711 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f19008bc700 1 -- 172.21.15.194:0/2087197991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18e00049c0 msgr2=0x7f18e0004e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:20.712 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f19008bc700 1 --2- 172.21.15.194:0/2087197991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18e00049c0 0x7f18e0004e40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f18f4009e60 tx=0x7f18f40091d0).stop 2021-03-09T09:38:20.712 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f18fad9d700 1 -- 172.21.15.194:0/2087197991 reap_dead start 2021-03-09T09:38:20.712 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f19008bc700 1 -- 172.21.15.194:0/2087197991 shutdown_connections 2021-03-09T09:38:20.712 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f19008bc700 1 -- 172.21.15.194:0/2087197991 >> 172.21.15.194:0/2087197991 conn(0x7f18fc0649c0 msgr2=0x7f18fc061c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:20.712 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f19008bc700 1 -- 172.21.15.194:0/2087197991 shutdown_connections 2021-03-09T09:38:20.713 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.699+0000 7f19008bc700 1 -- 172.21.15.194:0/2087197991 wait complete. 2021-03-09T09:38:20.713 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f19008bc700 1 Processor -- start 2021-03-09T09:38:20.715 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f19008bc700 1 -- start start 2021-03-09T09:38:20.716 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f19008bc700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc09fe20 0x7f18fc0a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.716 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f19008bc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18fc09e810 0x7f18fc09ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.716 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f19008bc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18fc14fbc0 0x7f18fc150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:20.716 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f19008bc700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc142340 con 0x7f18fc09e810 2021-03-09T09:38:20.717 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f19008bc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc1425e0 con 0x7f18fc14fbc0 2021-03-09T09:38:20.717 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f19008bc700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc142880 con 0x7f18fc09fe20 2021-03-09T09:38:20.717 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18fad9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18fc14fbc0 0x7f18fc150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.717 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18fad9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18fc14fbc0 0x7f18fc150040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:33166/0 (socket says 172.21.15.194:33166) 2021-03-09T09:38:20.718 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18fad9d700 1 -- 172.21.15.194:0/1141957002 learned_addr learned my addr 172.21.15.194:0/1141957002 (peer_addr_for_me v2:172.21.15.194:0/0) 2021-03-09T09:38:20.718 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18fa59c700 1 --2- 172.21.15.194:0/1141957002 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc09fe20 0x7f18fc0a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.718 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18f9d9b700 1 --2- 172.21.15.194:0/1141957002 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18fc09e810 0x7f18fc09ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:20.718 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18fad9d700 1 -- 172.21.15.194:0/1141957002 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc09fe20 msgr2=0x7f18fc0a0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:20.718 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18fad9d700 1 --2- 172.21.15.194:0/1141957002 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc09fe20 0x7f18fc0a0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.719 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18fad9d700 1 -- 172.21.15.194:0/1141957002 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18fc09e810 msgr2=0x7f18fc09ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:20.719 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18fad9d700 1 --2- 172.21.15.194:0/1141957002 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18fc09e810 0x7f18fc09ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.719 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18fad9d700 1 -- 172.21.15.194:0/1141957002 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc09d520 con 0x7f18fc14fbc0 2021-03-09T09:38:20.719 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18fad9d700 1 --2- 172.21.15.194:0/1141957002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18fc14fbc0 0x7f18fc150040 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f18f4014000 tx=0x7f18f401a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:20.720 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18ea7fc700 1 -- 172.21.15.194:0/1141957002 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f401c6c0 con 0x7f18fc14fbc0 2021-03-09T09:38:20.720 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18ea7fc700 1 -- 172.21.15.194:0/1141957002 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f401c820 con 0x7f18fc14fbc0 2021-03-09T09:38:20.720 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f19008bc700 1 -- 172.21.15.194:0/1141957002 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18fc144160 con 0x7f18fc14fbc0 2021-03-09T09:38:20.720 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18ea7fc700 1 -- 172.21.15.194:0/1141957002 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f401cbc0 con 0x7f18fc14fbc0 2021-03-09T09:38:20.720 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f19008bc700 1 -- 172.21.15.194:0/1141957002 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18fc1506d0 con 0x7f18fc14fbc0 2021-03-09T09:38:20.721 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18ea7fc700 1 -- 172.21.15.194:0/1141957002 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f18f4025440 con 0x7f18fc14fbc0 2021-03-09T09:38:20.721 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18ea7fc700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T09:38:20.721 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18ea7fc700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T09:38:20.721 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18ea7fc700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T09:38:20.721 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18ea7fc700 4 mgrc reconnect No active mgr available yet 2021-03-09T09:38:20.722 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.703+0000 7f18ea7fc700 1 -- 172.21.15.194:0/1141957002 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f18f4024750 con 0x7f18fc14fbc0 2021-03-09T09:38:20.722 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.707+0000 7f19008bc700 1 -- 172.21.15.194:0/1141957002 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18fc045aa0 con 0x7f18fc14fbc0 2021-03-09T09:38:20.722 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.711+0000 7f18ea7fc700 1 -- 172.21.15.194:0/1141957002 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f18f4016020 con 0x7f18fc14fbc0 2021-03-09T09:38:20.795 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.795+0000 7f19008bc700 1 -- 172.21.15.194:0/1141957002 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd new", "uuid": "76d971ae-5519-47c8-a3d6-668100f48891", "id": 0} v 0) v1 -- 0x7f18fc0a83a0 con 0x7f18fc14fbc0 2021-03-09T09:38:20.803 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f18ea7fc700 1 -- 172.21.15.194:0/1141957002 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76d971ae-5519-47c8-a3d6-668100f48891", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f18f4025810 con 0x7f18fc14fbc0 2021-03-09T09:38:20.804 INFO:teuthology.orchestra.run.smithi194.stdout:0 2021-03-09T09:38:20.805 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f19008bc700 10 mgrc shutdown 2021-03-09T09:38:20.805 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f19008bc700 1 -- 172.21.15.194:0/1141957002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18fc14fbc0 msgr2=0x7f18fc150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:20.806 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f19008bc700 1 --2- 172.21.15.194:0/1141957002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18fc14fbc0 0x7f18fc150040 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f18f4014000 tx=0x7f18f401a000).stop 2021-03-09T09:38:20.806 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f19008bc700 1 -- 172.21.15.194:0/1141957002 shutdown_connections 2021-03-09T09:38:20.806 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f19008bc700 1 --2- 172.21.15.194:0/1141957002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18fc14fbc0 0x7f18fc150040 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.806 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f19008bc700 1 --2- 172.21.15.194:0/1141957002 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f18fc09fe20 0x7f18fc0a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.806 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f19008bc700 1 --2- 172.21.15.194:0/1141957002 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18fc09e810 0x7f18fc09ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:20.807 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f19008bc700 1 -- 172.21.15.194:0/1141957002 >> 172.21.15.194:0/1141957002 conn(0x7f18fc0649c0 msgr2=0x7f18fc0631c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:20.807 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f19008bc700 1 -- 172.21.15.194:0/1141957002 shutdown_connections 2021-03-09T09:38:20.807 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.803+0000 7f19008bc700 1 -- 172.21.15.194:0/1141957002 wait complete. 2021-03-09T09:38:20.825 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new fe38e427-1abd-4b6c-a593-a5dc62b99540 1 2021-03-09T09:38:20.993 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9c4b5a700 1 Processor -- start 2021-03-09T09:38:20.993 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9c4b5a700 1 -- start start 2021-03-09T09:38:21.003 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9c4b5a700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c009dee0 0x7fb9c00a0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.003 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9c4b5a700 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c00a0620 con 0x7fb9c0067e00 2021-03-09T09:38:21.004 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9c4b5a700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c009e2c0 con 0x7fb9c0068240 2021-03-09T09:38:21.004 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9c4b5a700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00ab950 con 0x7fb9c009dee0 2021-03-09T09:38:21.004 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9bffff700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fb9c0068240 0x7fb9c013fe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:60556/0 (socket says 172.21.15.194:60556) 2021-03-09T09:38:21.005 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9bffff700 1 -- 172.21.15.194:0/281683920 learned_addr learned my addr 172.21.15.194:0/281683920 (peer_addr_for_me v1:172.21.15.194:0/0) 2021-03-09T09:38:21.005 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9bf7fe700 1 --2- 172.21.15.194:0/281683920 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c009dee0 0x7fb9c00a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9be7fc700 1 -- 172.21.15.194:0/281683920 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125341890 0 0) 0x7fb9c009e2c0 con 0x7fb9c0068240 2021-03-09T09:38:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.991+0000 7fb9be7fc700 1 -- 172.21.15.194:0/281683920 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ac003230 con 0x7fb9c0068240 2021-03-09T09:38:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9be7fc700 1 -- 172.21.15.194:0/281683920 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3182464035 0 0) 0x7fb9ac003230 con 0x7fb9c0068240 2021-03-09T09:38:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 -- 172.21.15.194:0/281683920 >> v1:172.21.15.194:6789/0 conn(0x7fb9c0068240 legacy=0x7fb9c013fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 -- 172.21.15.194:0/281683920 >> v1:172.21.15.52:6789/0 conn(0x7fb9c0067e00 legacy=0x7fb9c0134960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.007 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 -- 172.21.15.194:0/281683920 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00abbf0 con 0x7fb9c009dee0 2021-03-09T09:38:21.007 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 --2- 172.21.15.194:0/281683920 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c009dee0 0x7fb9c00a0130 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb9b000b920 tx=0x7fb9b0006d80).ready entity=mon.2 client_cookie=eb3a27ecc1415348 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9be7fc700 1 -- 172.21.15.194:0/281683920 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80018a0 con 0x7fb9c009dee0 2021-03-09T09:38:21.009 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9be7fc700 1 -- 172.21.15.194:0/281683920 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b0011400 con 0x7fb9c009dee0 2021-03-09T09:38:21.009 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9be7fc700 1 -- 172.21.15.194:0/281683920 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0011780 con 0x7fb9c009dee0 2021-03-09T09:38:21.009 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/281683920 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c009dee0 msgr2=0x7fb9c00a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.010 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 --2- 172.21.15.194:0/281683920 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c009dee0 0x7fb9c00a0130 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb9b000b920 tx=0x7fb9b0006d80).stop 2021-03-09T09:38:21.010 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/281683920 shutdown_connections 2021-03-09T09:38:21.010 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 --2- 172.21.15.194:0/281683920 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c009dee0 0x7fb9c00a0130 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.010 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/281683920 >> 172.21.15.194:0/281683920 conn(0x7fb9c0068e00 msgr2=0x7fb9c0061a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:21.011 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/281683920 shutdown_connections 2021-03-09T09:38:21.011 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/281683920 wait complete. 2021-03-09T09:38:21.011 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 Processor -- start 2021-03-09T09:38:21.011 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 -- start start 2021-03-09T09:38:21.011 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9c0067e00 0x7fb9c0138ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.012 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c0068240 0x7fb9c013a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.012 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9c009dee0 0x7fb9c013a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.013 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00a9ec0 con 0x7fb9c009dee0 2021-03-09T09:38:21.013 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c01483f0 con 0x7fb9c0067e00 2021-03-09T09:38:21.013 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9c4b5a700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c013f840 con 0x7fb9c0068240 2021-03-09T09:38:21.013 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9c0067e00 0x7fb9c0138ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.014 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9c0067e00 0x7fb9c0138ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:33174/0 (socket says 172.21.15.194:33174) 2021-03-09T09:38:21.014 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 -- 172.21.15.194:0/1278027715 learned_addr learned my addr 172.21.15.194:0/1278027715 (peer_addr_for_me v2:172.21.15.194:0/0) 2021-03-09T09:38:21.014 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9beffd700 1 --2- 172.21.15.194:0/1278027715 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c0068240 0x7fb9c013a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.014 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bffff700 1 --2- 172.21.15.194:0/1278027715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9c009dee0 0x7fb9c013a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.015 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 -- 172.21.15.194:0/1278027715 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c0068240 msgr2=0x7fb9c013a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.015 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 --2- 172.21.15.194:0/1278027715 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c0068240 0x7fb9c013a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.015 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 -- 172.21.15.194:0/1278027715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9c009dee0 msgr2=0x7fb9c013a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.016 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 --2- 172.21.15.194:0/1278027715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9c009dee0 0x7fb9c013a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.016 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 -- 172.21.15.194:0/1278027715 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00a7420 con 0x7fb9c0067e00 2021-03-09T09:38:21.016 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.995+0000 7fb9bf7fe700 1 --2- 172.21.15.194:0/1278027715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9c0067e00 0x7fb9c0138ef0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fb9b000b8f0 tx=0x7fb9b0017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:21.016 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9a77fe700 1 -- 172.21.15.194:0/1278027715 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0010a60 con 0x7fb9c0067e00 2021-03-09T09:38:21.017 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9a77fe700 1 -- 172.21.15.194:0/1278027715 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b0010bc0 con 0x7fb9c0067e00 2021-03-09T09:38:21.017 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/1278027715 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c00a7720 con 0x7fb9c0067e00 2021-03-09T09:38:21.017 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9a77fe700 1 -- 172.21.15.194:0/1278027715 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b001c420 con 0x7fb9c0067e00 2021-03-09T09:38:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/1278027715 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c01393e0 con 0x7fb9c0067e00 2021-03-09T09:38:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9a77fe700 1 -- 172.21.15.194:0/1278027715 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7fb9b000f070 con 0x7fb9c0067e00 2021-03-09T09:38:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9a77fe700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T09:38:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9a77fe700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T09:38:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9a77fe700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T09:38:21.019 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9a77fe700 4 mgrc reconnect No active mgr available yet 2021-03-09T09:38:21.019 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9a77fe700 1 -- 172.21.15.194:0/1278027715 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7fb9b0027460 con 0x7fb9c0067e00 2021-03-09T09:38:21.019 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:20.999+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/1278027715 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c00ab530 con 0x7fb9c0067e00 2021-03-09T09:38:21.020 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.003+0000 7fb9a77fe700 1 -- 172.21.15.194:0/1278027715 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7fb9c00a7720 con 0x7fb9c0067e00 2021-03-09T09:38:21.084 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.083+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/1278027715 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fe38e427-1abd-4b6c-a593-a5dc62b99540", "id": 1} v 0) v1 -- 0x7fb9c0045240 con 0x7fb9c0067e00 2021-03-09T09:38:21.091 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9a77fe700 1 -- 172.21.15.194:0/1278027715 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fe38e427-1abd-4b6c-a593-a5dc62b99540", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb9b001c810 con 0x7fb9c0067e00 2021-03-09T09:38:21.091 INFO:teuthology.orchestra.run.smithi194.stdout:1 2021-03-09T09:38:21.092 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9c4b5a700 10 mgrc shutdown 2021-03-09T09:38:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/1278027715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9c0067e00 msgr2=0x7fb9c0138ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9c4b5a700 1 --2- 172.21.15.194:0/1278027715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9c0067e00 0x7fb9c0138ef0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fb9b000b8f0 tx=0x7fb9b0017000).stop 2021-03-09T09:38:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/1278027715 shutdown_connections 2021-03-09T09:38:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9c4b5a700 1 --2- 172.21.15.194:0/1278027715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9c009dee0 0x7fb9c013a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9c4b5a700 1 --2- 172.21.15.194:0/1278027715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9c0067e00 0x7fb9c0138ef0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9c4b5a700 1 --2- 172.21.15.194:0/1278027715 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb9c0068240 0x7fb9c013a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/1278027715 >> 172.21.15.194:0/1278027715 conn(0x7fb9c0068e00 msgr2=0x7fb9c00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/1278027715 shutdown_connections 2021-03-09T09:38:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.091+0000 7fb9c4b5a700 1 -- 172.21.15.194:0/1278027715 wait complete. 2021-03-09T09:38:21.108 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 63b3755a-f879-4c75-b2dd-eeb09d99f7ac 2 2021-03-09T09:38:21.294 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23aa9fb700 1 Processor -- start 2021-03-09T09:38:21.295 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23aa9fb700 1 -- start start 2021-03-09T09:38:21.295 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23aa9fb700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a409f200 0x7f23a4137500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.295 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23aa9fb700 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f23a40a9e80 0x7f23a41379f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.296 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23aa9fb700 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a4061540 con 0x7f23a4061160 2021-03-09T09:38:21.296 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23aa9fb700 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a409e270 con 0x7f23a40a9e80 2021-03-09T09:38:21.296 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23aa9fb700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a409e510 con 0x7f23a409f200 2021-03-09T09:38:21.297 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a37fe700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a409f200 0x7f23a4137500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.297 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a8f98700 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f23a40a9e80 0x7f23a41379f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.297 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a37fe700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a409f200 0x7f23a4137500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3301/0 says I am v2:172.21.15.194:34438/0 (socket says 172.21.15.194:34438) 2021-03-09T09:38:21.297 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a8f98700 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f23a40a9e80 0x7f23a41379f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.194:46898/0 (socket says 172.21.15.194:46898) 2021-03-09T09:38:21.298 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a37fe700 1 -- 172.21.15.194:0/2458408344 learned_addr learned my addr 172.21.15.194:0/2458408344 (peer_addr_for_me v2:172.21.15.194:0/0) 2021-03-09T09:38:21.298 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a3fff700 1 --1- 172.21.15.194:0/2458408344 >> v1:172.21.15.52:6789/0 conn(0x7f23a4061160 0x7f23a409da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.194:36502/0 (socket says 172.21.15.194:36502) 2021-03-09T09:38:21.298 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a37fe700 1 -- 172.21.15.194:0/2458408344 >> v2:172.21.15.52:3300/0 conn(0x7f23a40a9e80 msgr2=0x7f23a41379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T09:38:21.298 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a37fe700 1 --2- 172.21.15.194:0/2458408344 >> v2:172.21.15.52:3300/0 conn(0x7f23a40a9e80 0x7f23a41379f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.298 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a37fe700 1 -- 172.21.15.194:0/2458408344 >> v1:172.21.15.52:6789/0 conn(0x7f23a4061160 legacy=0x7f23a409da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.299 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a37fe700 1 -- 172.21.15.194:0/2458408344 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40a9bc0 con 0x7f23a409f200 2021-03-09T09:38:21.299 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a8f98700 1 --2- 172.21.15.194:0/2458408344 >> v2:172.21.15.52:3300/0 conn(0x7f23a40a9e80 0x7f23a41379f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-03-09T09:38:21.299 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a37fe700 1 --2- 172.21.15.194:0/2458408344 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a409f200 0x7f23a4137500 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f239800b9e0 tx=0x7f2398006ea0).ready entity=mon.2 client_cookie=2bbf40fa8389a815 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:21.300 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a2ffd700 1 -- 172.21.15.194:0/2458408344 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23980098a0 con 0x7f23a409f200 2021-03-09T09:38:21.300 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a2ffd700 1 -- 172.21.15.194:0/2458408344 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2398009a00 con 0x7f23a409f200 2021-03-09T09:38:21.300 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.279+0000 7f23a2ffd700 1 -- 172.21.15.194:0/2458408344 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398009d80 con 0x7f23a409f200 2021-03-09T09:38:21.300 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 -- 172.21.15.194:0/2458408344 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a409f200 msgr2=0x7f23a4137500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.301 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 --2- 172.21.15.194:0/2458408344 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a409f200 0x7f23a4137500 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f239800b9e0 tx=0x7f2398006ea0).stop 2021-03-09T09:38:21.301 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 -- 172.21.15.194:0/2458408344 shutdown_connections 2021-03-09T09:38:21.301 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 --2- 172.21.15.194:0/2458408344 >> v2:172.21.15.52:3300/0 conn(0x7f23a40a9e80 0x7f23a41379f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.301 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 --2- 172.21.15.194:0/2458408344 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a409f200 0x7f23a4137500 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.302 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 -- 172.21.15.194:0/2458408344 >> 172.21.15.194:0/2458408344 conn(0x7f23a40628b0 msgr2=0x7f23a4064fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:21.302 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 -- 172.21.15.194:0/2458408344 shutdown_connections 2021-03-09T09:38:21.302 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 -- 172.21.15.194:0/2458408344 wait complete. 2021-03-09T09:38:21.302 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 Processor -- start 2021-03-09T09:38:21.302 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 -- start start 2021-03-09T09:38:21.303 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a4061160 0x7f23a40a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.303 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a409f200 0x7f23a408ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.303 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23a40a9e80 0x7f23a414bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.303 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40a7510 con 0x7f23a40a9e80 2021-03-09T09:38:21.303 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a408f470 con 0x7f23a409f200 2021-03-09T09:38:21.304 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23aa9fb700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40a7da0 con 0x7f23a4061160 2021-03-09T09:38:21.304 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23a37fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a409f200 0x7f23a408ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.304 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23a37fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a409f200 0x7f23a408ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:33188/0 (socket says 172.21.15.194:33188) 2021-03-09T09:38:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23a37fe700 1 -- 172.21.15.194:0/3410766929 learned_addr learned my addr 172.21.15.194:0/3410766929 (peer_addr_for_me v2:172.21.15.194:0/0) 2021-03-09T09:38:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23a3fff700 1 --2- 172.21.15.194:0/3410766929 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a4061160 0x7f23a40a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23a8f98700 1 --2- 172.21.15.194:0/3410766929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23a40a9e80 0x7f23a414bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23a37fe700 1 -- 172.21.15.194:0/3410766929 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a4061160 msgr2=0x7f23a40a7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23a37fe700 1 --2- 172.21.15.194:0/3410766929 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a4061160 0x7f23a40a7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.306 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23a37fe700 1 -- 172.21.15.194:0/3410766929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23a40a9e80 msgr2=0x7f23a414bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.306 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23a37fe700 1 --2- 172.21.15.194:0/3410766929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23a40a9e80 0x7f23a414bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.306 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.283+0000 7f23a37fe700 1 -- 172.21.15.194:0/3410766929 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a413d470 con 0x7f23a409f200 2021-03-09T09:38:21.307 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f23a37fe700 1 --2- 172.21.15.194:0/3410766929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a409f200 0x7f23a408ea70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f2398006ee0 tx=0x7f2398016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:21.307 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f238bfff700 1 -- 172.21.15.194:0/3410766929 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398012d60 con 0x7f23a409f200 2021-03-09T09:38:21.307 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f238bfff700 1 -- 172.21.15.194:0/3410766929 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239801c420 con 0x7f23a409f200 2021-03-09T09:38:21.307 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f23aa9fb700 1 -- 172.21.15.194:0/3410766929 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a4138160 con 0x7f23a409f200 2021-03-09T09:38:21.308 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f238bfff700 1 -- 172.21.15.194:0/3410766929 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239801c7e0 con 0x7f23a409f200 2021-03-09T09:38:21.308 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f23aa9fb700 1 -- 172.21.15.194:0/3410766929 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a408f1d0 con 0x7f23a409f200 2021-03-09T09:38:21.308 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f238bfff700 1 -- 172.21.15.194:0/3410766929 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f239800f070 con 0x7f23a409f200 2021-03-09T09:38:21.308 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f238bfff700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T09:38:21.308 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f238bfff700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T09:38:21.309 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f238bfff700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T09:38:21.309 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f238bfff700 4 mgrc reconnect No active mgr available yet 2021-03-09T09:38:21.309 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f238bfff700 1 -- 172.21.15.194:0/3410766929 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7f2398026ca0 con 0x7f23a409f200 2021-03-09T09:38:21.309 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.287+0000 7f23aa9fb700 1 -- 172.21.15.194:0/3410766929 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a4066120 con 0x7f23a409f200 2021-03-09T09:38:21.309 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.291+0000 7f238bfff700 1 -- 172.21.15.194:0/3410766929 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f2398016070 con 0x7f23a409f200 2021-03-09T09:38:21.384 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.383+0000 7f23aa9fb700 1 -- 172.21.15.194:0/3410766929 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd new", "uuid": "63b3755a-f879-4c75-b2dd-eeb09d99f7ac", "id": 2} v 0) v1 -- 0x7f23a4045aa0 con 0x7f23a409f200 2021-03-09T09:38:21.391 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f238bfff700 1 -- 172.21.15.194:0/3410766929 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "63b3755a-f879-4c75-b2dd-eeb09d99f7ac", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f239800b190 con 0x7f23a409f200 2021-03-09T09:38:21.392 INFO:teuthology.orchestra.run.smithi194.stdout:2 2021-03-09T09:38:21.393 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f23aa9fb700 10 mgrc shutdown 2021-03-09T09:38:21.394 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f23aa9fb700 1 -- 172.21.15.194:0/3410766929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a409f200 msgr2=0x7f23a408ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.394 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f23aa9fb700 1 --2- 172.21.15.194:0/3410766929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a409f200 0x7f23a408ea70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f2398006ee0 tx=0x7f2398016000).stop 2021-03-09T09:38:21.394 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f23aa9fb700 1 -- 172.21.15.194:0/3410766929 shutdown_connections 2021-03-09T09:38:21.395 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f23aa9fb700 1 --2- 172.21.15.194:0/3410766929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23a40a9e80 0x7f23a414bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.395 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f23aa9fb700 1 --2- 172.21.15.194:0/3410766929 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f23a4061160 0x7f23a40a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.395 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f23aa9fb700 1 --2- 172.21.15.194:0/3410766929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a409f200 0x7f23a408ea70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.395 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f23aa9fb700 1 -- 172.21.15.194:0/3410766929 >> 172.21.15.194:0/3410766929 conn(0x7f23a40628b0 msgr2=0x7f23a40abd00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:21.395 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f23aa9fb700 1 -- 172.21.15.194:0/3410766929 shutdown_connections 2021-03-09T09:38:21.396 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.391+0000 7f23aa9fb700 1 -- 172.21.15.194:0/3410766929 wait complete. 2021-03-09T09:38:21.415 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new b9634dd7-a1fe-4bbd-950a-af1aba0e19fa 3 2021-03-09T09:38:21.579 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 Processor -- start 2021-03-09T09:38:21.579 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 -- start start 2021-03-09T09:38:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb34005ca90 0x7fb340068210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.591 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fb340138340 0x7fb340138790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.592 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb340068700 con 0x7fb340066350 2021-03-09T09:38:21.592 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb340067cf0 con 0x7fb340138340 2021-03-09T09:38:21.593 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3400a94d0 con 0x7fb34005ca90 2021-03-09T09:38:21.593 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb346003700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb34005ca90 0x7fb340068210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.594 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb346003700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb34005ca90 0x7fb340068210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:33192/0 (socket says 172.21.15.194:33192) 2021-03-09T09:38:21.594 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb346003700 1 -- 172.21.15.194:0/899421972 learned_addr learned my addr 172.21.15.194:0/899421972 (peer_addr_for_me v2:172.21.15.194:0/0) 2021-03-09T09:38:21.594 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb346804700 1 --2- 172.21.15.194:0/899421972 >> v2:172.21.15.52:3300/0 conn(0x7fb340138340 0x7fb340138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.595 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb346003700 1 -- 172.21.15.194:0/899421972 >> v2:172.21.15.52:3300/0 conn(0x7fb340138340 msgr2=0x7fb340138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T09:38:21.595 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb346003700 1 --2- 172.21.15.194:0/899421972 >> v2:172.21.15.52:3300/0 conn(0x7fb340138340 0x7fb340138790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.595 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb346003700 1 -- 172.21.15.194:0/899421972 >> v1:172.21.15.52:6789/0 conn(0x7fb340066350 legacy=0x7fb340067270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T09:38:21.596 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb346003700 1 -- 172.21.15.194:0/899421972 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34005ce70 con 0x7fb34005ca90 2021-03-09T09:38:21.596 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb346003700 1 --2- 172.21.15.194:0/899421972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 0x7fb340068210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb340062fc0 tx=0x7fb33c006ea0).ready entity=mon.1 client_cookie=3e46ccd2bb47f12c server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:21.596 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb345001700 1 -- 172.21.15.194:0/899421972 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c009890 con 0x7fb34005ca90 2021-03-09T09:38:21.597 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb345001700 1 -- 172.21.15.194:0/899421972 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c0099f0 con 0x7fb34005ca90 2021-03-09T09:38:21.597 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb345001700 1 -- 172.21.15.194:0/899421972 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c009d70 con 0x7fb34005ca90 2021-03-09T09:38:21.597 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 -- 172.21.15.194:0/899421972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 msgr2=0x7fb340068210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.598 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 --2- 172.21.15.194:0/899421972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 0x7fb340068210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb340062fc0 tx=0x7fb33c006ea0).stop 2021-03-09T09:38:21.598 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 -- 172.21.15.194:0/899421972 shutdown_connections 2021-03-09T09:38:21.598 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 --2- 172.21.15.194:0/899421972 >> v2:172.21.15.52:3300/0 conn(0x7fb340138340 0x7fb340138790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 --2- 172.21.15.194:0/899421972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 0x7fb340068210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 -- 172.21.15.194:0/899421972 >> 172.21.15.194:0/899421972 conn(0x7fb3400649c0 msgr2=0x7fb340068e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 -- 172.21.15.194:0/899421972 shutdown_connections 2021-03-09T09:38:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 -- 172.21.15.194:0/899421972 wait complete. 2021-03-09T09:38:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 Processor -- start 2021-03-09T09:38:21.600 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.579+0000 7fb347005700 1 -- start start 2021-03-09T09:38:21.600 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb347005700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 0x7fb3400a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.600 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb347005700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb340066350 0x7fb340138790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.601 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb347005700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb340138340 0x7fb3401392f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:21.601 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb347005700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3400a8830 con 0x7fb340066350 2021-03-09T09:38:21.601 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb347005700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb340139050 con 0x7fb34005ca90 2021-03-09T09:38:21.602 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb347005700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34013a4e0 con 0x7fb340138340 2021-03-09T09:38:21.602 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb346003700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 0x7fb3400a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.602 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb346003700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 0x7fb3400a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:33198/0 (socket says 172.21.15.194:33198) 2021-03-09T09:38:21.603 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb346003700 1 -- 172.21.15.194:0/1345203584 learned_addr learned my addr 172.21.15.194:0/1345203584 (peer_addr_for_me v2:172.21.15.194:0/0) 2021-03-09T09:38:21.603 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb346804700 1 --2- 172.21.15.194:0/1345203584 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb340138340 0x7fb3401392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.603 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb346003700 1 -- 172.21.15.194:0/1345203584 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb340138340 msgr2=0x7fb3401392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.604 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb346003700 1 --2- 172.21.15.194:0/1345203584 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb340138340 0x7fb3401392f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.604 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb346003700 1 -- 172.21.15.194:0/1345203584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb340066350 msgr2=0x7fb340138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.604 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb345802700 1 --2- 172.21.15.194:0/1345203584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb340066350 0x7fb340138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:21.604 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb346003700 1 --2- 172.21.15.194:0/1345203584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb340066350 0x7fb340138790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.605 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb346003700 1 -- 172.21.15.194:0/1345203584 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3400ac7e0 con 0x7fb34005ca90 2021-03-09T09:38:21.605 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb346003700 1 --2- 172.21.15.194:0/1345203584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 0x7fb3400a1a40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fb33c006ee0 tx=0x7fb33c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:21.605 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb32e7fc700 1 -- 172.21.15.194:0/1345203584 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c012e40 con 0x7fb34005ca90 2021-03-09T09:38:21.606 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb32e7fc700 1 -- 172.21.15.194:0/1345203584 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c01c420 con 0x7fb34005ca90 2021-03-09T09:38:21.606 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb347005700 1 -- 172.21.15.194:0/1345203584 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb340138c80 con 0x7fb34005ca90 2021-03-09T09:38:21.606 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb32e7fc700 1 -- 172.21.15.194:0/1345203584 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c01c7e0 con 0x7fb34005ca90 2021-03-09T09:38:21.606 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb347005700 1 -- 172.21.15.194:0/1345203584 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3400a3580 con 0x7fb34005ca90 2021-03-09T09:38:21.607 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb32e7fc700 1 -- 172.21.15.194:0/1345203584 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7fb33c00f070 con 0x7fb34005ca90 2021-03-09T09:38:21.607 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb32e7fc700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T09:38:21.607 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb32e7fc700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T09:38:21.607 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb32e7fc700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T09:38:21.608 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb32e7fc700 4 mgrc reconnect No active mgr available yet 2021-03-09T09:38:21.608 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb32e7fc700 1 -- 172.21.15.194:0/1345203584 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7fb33c01b380 con 0x7fb34005ca90 2021-03-09T09:38:21.608 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.583+0000 7fb347005700 1 -- 172.21.15.194:0/1345203584 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34009cca0 con 0x7fb34005ca90 2021-03-09T09:38:21.608 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.587+0000 7fb32e7fc700 1 -- 172.21.15.194:0/1345203584 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7fb33c016070 con 0x7fb34005ca90 2021-03-09T09:38:21.685 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.683+0000 7fb347005700 1 -- 172.21.15.194:0/1345203584 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b9634dd7-a1fe-4bbd-950a-af1aba0e19fa", "id": 3} v 0) v1 -- 0x7fb340045240 con 0x7fb34005ca90 2021-03-09T09:38:21.692 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb32e7fc700 1 -- 172.21.15.194:0/1345203584 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b9634dd7-a1fe-4bbd-950a-af1aba0e19fa", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb33c006950 con 0x7fb34005ca90 2021-03-09T09:38:21.692 INFO:teuthology.orchestra.run.smithi194.stdout:3 2021-03-09T09:38:21.694 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb347005700 10 mgrc shutdown 2021-03-09T09:38:21.694 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb347005700 1 -- 172.21.15.194:0/1345203584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 msgr2=0x7fb3400a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:21.694 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb347005700 1 --2- 172.21.15.194:0/1345203584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 0x7fb3400a1a40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fb33c006ee0 tx=0x7fb33c016000).stop 2021-03-09T09:38:21.695 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb347005700 1 -- 172.21.15.194:0/1345203584 shutdown_connections 2021-03-09T09:38:21.695 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb347005700 1 --2- 172.21.15.194:0/1345203584 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fb340138340 0x7fb3401392f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.695 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb347005700 1 --2- 172.21.15.194:0/1345203584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34005ca90 0x7fb3400a1a40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.695 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb347005700 1 --2- 172.21.15.194:0/1345203584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb340066350 0x7fb340138790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:21.696 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb347005700 1 -- 172.21.15.194:0/1345203584 >> 172.21.15.194:0/1345203584 conn(0x7fb3400649c0 msgr2=0x7fb34013eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:21.696 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb347005700 1 -- 172.21.15.194:0/1345203584 shutdown_connections 2021-03-09T09:38:21.696 INFO:teuthology.orchestra.run.smithi194.stderr:2021-03-09T09:38:21.691+0000 7fb347005700 1 -- 172.21.15.194:0/1345203584 wait complete. 2021-03-09T09:38:21.711 INFO:tasks.ceph.osd.0:Restarting daemon 2021-03-09T09:38:21.712 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-03-09T09:38:21.714 INFO:tasks.ceph.osd.0:Started 2021-03-09T09:38:21.715 INFO:tasks.ceph.osd.1:Restarting daemon 2021-03-09T09:38:21.715 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-03-09T09:38:21.717 INFO:tasks.ceph.osd.1:Started 2021-03-09T09:38:21.718 INFO:tasks.ceph.osd.2:Restarting daemon 2021-03-09T09:38:21.718 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-03-09T09:38:21.720 INFO:tasks.ceph.osd.2:Started 2021-03-09T09:38:21.720 INFO:tasks.ceph.osd.3:Restarting daemon 2021-03-09T09:38:21.721 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-03-09T09:38:21.723 INFO:tasks.ceph.osd.3:Started 2021-03-09T09:38:21.723 INFO:tasks.ceph:Waiting for OSDs to come up 2021-03-09T09:38:22.127 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T09:38:22.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 Processor -- start 2021-03-09T09:38:22.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 -- start start 2021-03-09T09:38:22.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d480a7f10 0x7f7d480a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:22.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f7d48066720 0x7f7d480a9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:22.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d48066b60 con 0x7f7d48135160 2021-03-09T09:38:22.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d48066de0 con 0x7f7d48066720 2021-03-09T09:38:22.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d48067080 con 0x7f7d480a7f10 2021-03-09T09:38:22.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d47fff700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d480a7f10 0x7f7d480a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:22.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d47fff700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d480a7f10 0x7f7d480a8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3301/0 says I am v2:172.21.15.52:35034/0 (socket says 172.21.15.52:35034) 2021-03-09T09:38:22.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d47fff700 1 -- 172.21.15.52:0/1495746939 learned_addr learned my addr 172.21.15.52:0/1495746939 (peer_addr_for_me v2:172.21.15.52:0/0) 2021-03-09T09:38:22.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d477fe700 1 --2- 172.21.15.52:0/1495746939 >> v2:172.21.15.194:3300/0 conn(0x7f7d48066720 0x7f7d480a9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:22.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d46ffd700 1 -- 172.21.15.52:0/1495746939 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184855898 0 0) 0x7f7d48066b60 con 0x7f7d48135160 2021-03-09T09:38:22.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d46ffd700 1 -- 172.21.15.52:0/1495746939 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d28003230 con 0x7f7d48135160 2021-03-09T09:38:22.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d47fff700 1 -- 172.21.15.52:0/1495746939 >> v2:172.21.15.194:3300/0 conn(0x7f7d48066720 msgr2=0x7f7d480a9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T09:38:22.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d47fff700 1 --2- 172.21.15.52:0/1495746939 >> v2:172.21.15.194:3300/0 conn(0x7f7d48066720 0x7f7d480a9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:22.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d47fff700 1 -- 172.21.15.52:0/1495746939 >> v1:172.21.15.52:6789/0 conn(0x7f7d48135160 legacy=0x7f7d48135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:22.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d46ffd700 1 -- 172.21.15.52:0/1495746939 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1091988041 0 0) 0x7f7d28003230 con 0x7f7d48135160 2021-03-09T09:38:22.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d47fff700 1 -- 172.21.15.52:0/1495746939 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480a82f0 con 0x7f7d480a7f10 2021-03-09T09:38:22.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d47fff700 1 --2- 172.21.15.52:0/1495746939 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d480a7f10 0x7f7d480a8990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f7d3800b9e0 tx=0x7f7d38006ea0).ready entity=mon.2 client_cookie=46442c4f32628a82 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:22.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d46ffd700 1 -- 172.21.15.52:0/1495746939 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d400014e0 con 0x7f7d480a7f10 2021-03-09T09:38:22.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d46ffd700 1 -- 172.21.15.52:0/1495746939 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d380094a0 con 0x7f7d480a7f10 2021-03-09T09:38:22.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d46ffd700 1 -- 172.21.15.52:0/1495746939 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d38009820 con 0x7f7d480a7f10 2021-03-09T09:38:22.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 -- 172.21.15.52:0/1495746939 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d480a7f10 msgr2=0x7f7d480a8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:22.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 --2- 172.21.15.52:0/1495746939 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d480a7f10 0x7f7d480a8990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f7d3800b9e0 tx=0x7f7d38006ea0).stop 2021-03-09T09:38:22.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 -- 172.21.15.52:0/1495746939 shutdown_connections 2021-03-09T09:38:22.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 --2- 172.21.15.52:0/1495746939 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d480a7f10 0x7f7d480a8990 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:22.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 --2- 172.21.15.52:0/1495746939 >> v2:172.21.15.194:3300/0 conn(0x7f7d48066720 0x7f7d480a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:22.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 -- 172.21.15.52:0/1495746939 >> 172.21.15.52:0/1495746939 conn(0x7f7d48064b50 msgr2=0x7f7d48064f30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:22.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 -- 172.21.15.52:0/1495746939 shutdown_connections 2021-03-09T09:38:22.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.287+0000 7f7d4d742700 1 -- 172.21.15.52:0/1495746939 wait complete. 2021-03-09T09:38:22.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 Processor -- start 2021-03-09T09:38:22.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 -- start start 2021-03-09T09:38:22.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d48066720 0x7f7d480a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:22.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d480a7f10 0x7f7d480635a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:22.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d48135160 0x7f7d480a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:22.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4809f170 con 0x7f7d480a7f10 2021-03-09T09:38:22.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4809fbc0 con 0x7f7d48066720 2021-03-09T09:38:22.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4809fe60 con 0x7f7d48135160 2021-03-09T09:38:22.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4cf41700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d48135160 0x7f7d480a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:22.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4cf41700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d48135160 0x7f7d480a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3301/0 says I am v2:172.21.15.52:35042/0 (socket says 172.21.15.52:35042) 2021-03-09T09:38:22.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4cf41700 1 -- 172.21.15.52:0/401268500 learned_addr learned my addr 172.21.15.52:0/401268500 (peer_addr_for_me v2:172.21.15.52:0/0) 2021-03-09T09:38:22.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d477fe700 1 --2- 172.21.15.52:0/401268500 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d480a7f10 0x7f7d480635a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:22.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d47fff700 1 --2- 172.21.15.52:0/401268500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d48066720 0x7f7d480a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:22.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d47fff700 1 -- 172.21.15.52:0/401268500 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d48135160 msgr2=0x7f7d480a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:22.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d47fff700 1 --2- 172.21.15.52:0/401268500 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d48135160 0x7f7d480a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:22.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d47fff700 1 -- 172.21.15.52:0/401268500 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d480a7f10 msgr2=0x7f7d480635a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:22.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d47fff700 1 --2- 172.21.15.52:0/401268500 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d480a7f10 0x7f7d480635a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:22.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d47fff700 1 -- 172.21.15.52:0/401268500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480a18b0 con 0x7f7d48066720 2021-03-09T09:38:22.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d47fff700 1 --2- 172.21.15.52:0/401268500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d48066720 0x7f7d480a1f50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f7d3800b9e0 tx=0x7f7d38016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:22.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d3ffff700 1 -- 172.21.15.52:0/401268500 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d38011990 con 0x7f7d48066720 2021-03-09T09:38:22.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d3ffff700 1 -- 172.21.15.52:0/401268500 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d38011af0 con 0x7f7d48066720 2021-03-09T09:38:22.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 -- 172.21.15.52:0/401268500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d480a1b00 con 0x7f7d48066720 2021-03-09T09:38:22.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d3ffff700 1 -- 172.21.15.52:0/401268500 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d38011e90 con 0x7f7d48066720 2021-03-09T09:38:22.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 -- 172.21.15.52:0/401268500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4809f6b0 con 0x7f7d48066720 2021-03-09T09:38:22.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d3ffff700 1 -- 172.21.15.52:0/401268500 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f7d3800f070 con 0x7f7d48066720 2021-03-09T09:38:22.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d3ffff700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T09:38:22.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d3ffff700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T09:38:22.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d3ffff700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T09:38:22.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d3ffff700 4 mgrc reconnect No active mgr available yet 2021-03-09T09:38:22.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d3ffff700 1 -- 172.21.15.52:0/401268500 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7f7d3801dd20 con 0x7f7d48066720 2021-03-09T09:38:22.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.291+0000 7f7d4d742700 1 -- 172.21.15.52:0/401268500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4809f960 con 0x7f7d48066720 2021-03-09T09:38:22.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.299+0000 7f7d3ffff700 1 -- 172.21.15.52:0/401268500 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f7d38016070 con 0x7f7d48066720 2021-03-09T09:38:22.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.371+0000 7f7d4d742700 1 -- 172.21.15.52:0/401268500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d480589b0 con 0x7f7d48066720 2021-03-09T09:38:22.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.371+0000 7f7d3ffff700 1 -- 172.21.15.52:0/401268500 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4118 (secure 0 0 0) 0x7f7d3801d640 con 0x7f7d48066720 2021-03-09T09:38:22.376 INFO:teuthology.misc.health.smithi052.stdout: 2021-03-09T09:38:22.376 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":6,"fsid":"1325f083-8043-46e9-93ad-0afef33c673d","created":"2021-03-09T09:38:19.242162+0000","modified":"2021-03-09T09:38:21.688382+0000","last_up_change":"0.000000","last_in_change":"2021-03-09T09:38:21.688382+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"76d971ae-5519-47c8-a3d6-668100f48891","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fe38e427-1abd-4b6c-a593-a5dc62b99540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"63b3755a-f879-4c75-b2dd-eeb09d99f7ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b9634dd7-a1fe-4bbd-950a-af1aba0e19fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-09T09:38:22.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.375+0000 7f7d4d742700 10 mgrc shutdown 2021-03-09T09:38:22.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.375+0000 7f7d4d742700 1 -- 172.21.15.52:0/401268500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d48066720 msgr2=0x7f7d480a1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:22.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.375+0000 7f7d4d742700 1 --2- 172.21.15.52:0/401268500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d48066720 0x7f7d480a1f50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f7d3800b9e0 tx=0x7f7d38016000).stop 2021-03-09T09:38:22.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.375+0000 7f7d4d742700 1 -- 172.21.15.52:0/401268500 shutdown_connections 2021-03-09T09:38:22.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.375+0000 7f7d4d742700 1 --2- 172.21.15.52:0/401268500 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f7d48135160 0x7f7d480a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:22.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.375+0000 7f7d4d742700 1 --2- 172.21.15.52:0/401268500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d48066720 0x7f7d480a1f50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:22.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.375+0000 7f7d4d742700 1 --2- 172.21.15.52:0/401268500 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d480a7f10 0x7f7d480635a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:22.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.375+0000 7f7d4d742700 1 -- 172.21.15.52:0/401268500 >> 172.21.15.52:0/401268500 conn(0x7f7d48064b50 msgr2=0x7f7d48064f30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:22.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.375+0000 7f7d4d742700 1 -- 172.21.15.52:0/401268500 shutdown_connections 2021-03-09T09:38:22.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:22.375+0000 7f7d4d742700 1 -- 172.21.15.52:0/401268500 wait complete. 2021-03-09T09:38:22.386 INFO:tasks.ceph.osd.0.smithi052.stderr:2021-03-09T09:38:22.383+0000 7f7861ac8f00 -1 Falling back to public interface 2021-03-09T09:38:22.392 DEBUG:teuthology.misc:0 of 4 OSDs are up 2021-03-09T09:38:22.401 INFO:tasks.ceph.osd.3.smithi194.stderr:2021-03-09T09:38:22.399+0000 7fbc4d7dcf00 -1 Falling back to public interface 2021-03-09T09:38:22.435 INFO:tasks.ceph.osd.1.smithi194.stderr:2021-03-09T09:38:22.435+0000 7f2882720f00 -1 Falling back to public interface 2021-03-09T09:38:22.439 INFO:tasks.ceph.osd.2.smithi194.stderr:2021-03-09T09:38:22.439+0000 7fc57ea52f00 -1 Falling back to public interface 2021-03-09T09:38:24.071 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:38:24.074 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:38:26.087 INFO:tasks.ceph.osd.0.smithi052.stderr:2021-03-09T09:38:26.083+0000 7f7861ac8f00 -1 osd.0 0 log_to_monitors {default=true} 2021-03-09T09:38:26.090 INFO:tasks.ceph.osd.1.smithi194.stderr:2021-03-09T09:38:26.087+0000 7f2882720f00 -1 osd.1 0 log_to_monitors {default=true} 2021-03-09T09:38:26.248 INFO:tasks.ceph.osd.3.smithi194.stderr:2021-03-09T09:38:26.247+0000 7fbc4d7dcf00 -1 osd.3 0 log_to_monitors {default=true} 2021-03-09T09:38:26.300 INFO:tasks.ceph.osd.2.smithi194.stderr:2021-03-09T09:38:26.299+0000 7fc57ea52f00 -1 osd.2 0 log_to_monitors {default=true} 2021-03-09T09:38:27.210 INFO:tasks.ceph.osd.0.smithi052.stderr:2021-03-09T09:38:27.207+0000 7f7858f92700 -1 osd.0 0 waiting for initial osdmap 2021-03-09T09:38:27.212 INFO:tasks.ceph.osd.1.smithi194.stderr:2021-03-09T09:38:27.211+0000 7f2879bea700 -1 osd.1 0 waiting for initial osdmap 2021-03-09T09:38:28.226 INFO:tasks.ceph.osd.3.smithi194.stderr:2021-03-09T09:38:28.223+0000 7fbc44ca6700 -1 osd.3 0 waiting for initial osdmap 2021-03-09T09:38:28.227 INFO:tasks.ceph.osd.2.smithi194.stderr:2021-03-09T09:38:28.227+0000 7fc575f1c700 -1 osd.2 0 waiting for initial osdmap 2021-03-09T09:38:28.288 INFO:tasks.ceph.mon.a.smithi052.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc59f330700 time 2021-03-09T09:38:28.290285+0000 2021-03-09T09:38:28.289 INFO:tasks.ceph.mon.a.smithi052.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:38:28.301 INFO:tasks.ceph.mon.a.smithi052.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:28.301 INFO:tasks.ceph.mon.a.smithi052.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fc5abaf23dd] 2021-03-09T09:38:28.301 INFO:tasks.ceph.mon.a.smithi052.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fc5abaf25b8] 2021-03-09T09:38:28.301 INFO:tasks.ceph.mon.a.smithi052.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55e374d6da65] 2021-03-09T09:38:28.302 INFO:tasks.ceph.mon.a.smithi052.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55e374db9a71] 2021-03-09T09:38:28.302 INFO:tasks.ceph.mon.a.smithi052.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55e374de9295] 2021-03-09T09:38:28.302 INFO:tasks.ceph.mon.a.smithi052.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55e374df124b] 2021-03-09T09:38:28.303 INFO:tasks.ceph.mon.a.smithi052.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55e374df17ab] 2021-03-09T09:38:28.303 INFO:tasks.ceph.mon.a.smithi052.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55e374d60ade] 2021-03-09T09:38:28.303 INFO:tasks.ceph.mon.a.smithi052.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55e374c4bc79] 2021-03-09T09:38:28.303 INFO:tasks.ceph.mon.a.smithi052.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55e374c4f229] 2021-03-09T09:38:28.303 INFO:tasks.ceph.mon.a.smithi052.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.304 INFO:tasks.ceph.mon.a.smithi052.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55e374c5187f] 2021-03-09T09:38:28.304 INFO:tasks.ceph.mon.a.smithi052.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55e374c4fd70] 2021-03-09T09:38:28.304 INFO:tasks.ceph.mon.a.smithi052.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.304 INFO:tasks.ceph.mon.a.smithi052.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55e374c7e178] 2021-03-09T09:38:28.304 INFO:tasks.ceph.mon.a.smithi052.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7fc5abcf42ca] 2021-03-09T09:38:28.305 INFO:tasks.ceph.mon.a.smithi052.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fc5abd9e71d] 2021-03-09T09:38:28.305 INFO:tasks.ceph.mon.a.smithi052.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc5aa9626db] 2021-03-09T09:38:28.305 INFO:tasks.ceph.mon.a.smithi052.stderr: 19: clone() 2021-03-09T09:38:28.305 INFO:tasks.ceph.mon.a.smithi052.stderr:*** Caught signal (Aborted) ** 2021-03-09T09:38:28.306 INFO:tasks.ceph.mon.a.smithi052.stderr: in thread 7fc59f330700 thread_name:ms_dispatch 2021-03-09T09:38:28.306 INFO:tasks.ceph.mon.a.smithi052.stderr:2021-03-09T09:38:28.299+0000 7fc59f330700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc59f330700 time 2021-03-09T09:38:28.290285+0000 2021-03-09T09:38:28.306 INFO:tasks.ceph.mon.a.smithi052.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:38:28.306 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.307 INFO:tasks.ceph.mon.a.smithi052.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:28.307 INFO:tasks.ceph.mon.a.smithi052.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fc5abaf23dd] 2021-03-09T09:38:28.307 INFO:tasks.ceph.mon.a.smithi052.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fc5abaf25b8] 2021-03-09T09:38:28.307 INFO:tasks.ceph.mon.a.smithi052.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55e374d6da65] 2021-03-09T09:38:28.308 INFO:tasks.ceph.mon.a.smithi052.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55e374db9a71] 2021-03-09T09:38:28.308 INFO:tasks.ceph.mon.a.smithi052.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55e374de9295] 2021-03-09T09:38:28.308 INFO:tasks.ceph.mon.a.smithi052.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55e374df124b] 2021-03-09T09:38:28.308 INFO:tasks.ceph.mon.a.smithi052.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55e374df17ab] 2021-03-09T09:38:28.309 INFO:tasks.ceph.mon.a.smithi052.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55e374d60ade] 2021-03-09T09:38:28.309 INFO:tasks.ceph.mon.a.smithi052.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55e374c4bc79] 2021-03-09T09:38:28.309 INFO:tasks.ceph.mon.a.smithi052.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55e374c4f229] 2021-03-09T09:38:28.309 INFO:tasks.ceph.mon.a.smithi052.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.310 INFO:tasks.ceph.mon.a.smithi052.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55e374c5187f] 2021-03-09T09:38:28.310 INFO:tasks.ceph.mon.a.smithi052.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55e374c4fd70] 2021-03-09T09:38:28.310 INFO:tasks.ceph.mon.a.smithi052.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.310 INFO:tasks.ceph.mon.a.smithi052.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55e374c7e178] 2021-03-09T09:38:28.311 INFO:tasks.ceph.mon.a.smithi052.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7fc5abcf42ca] 2021-03-09T09:38:28.311 INFO:tasks.ceph.mon.a.smithi052.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fc5abd9e71d] 2021-03-09T09:38:28.311 INFO:tasks.ceph.mon.a.smithi052.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc5aa9626db] 2021-03-09T09:38:28.312 INFO:tasks.ceph.mon.a.smithi052.stderr: 19: clone() 2021-03-09T09:38:28.312 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.313 INFO:tasks.ceph.mon.a.smithi052.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:28.313 INFO:tasks.ceph.mon.a.smithi052.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fc5aa96d980] 2021-03-09T09:38:28.313 INFO:tasks.ceph.mon.a.smithi052.stderr: 2: gsignal() 2021-03-09T09:38:28.314 INFO:tasks.ceph.mon.a.smithi052.stderr: 3: abort() 2021-03-09T09:38:28.314 INFO:tasks.ceph.mon.a.smithi052.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fc5abaf242e] 2021-03-09T09:38:28.314 INFO:tasks.ceph.mon.a.smithi052.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fc5abaf25b8] 2021-03-09T09:38:28.315 INFO:tasks.ceph.mon.a.smithi052.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55e374d6da65] 2021-03-09T09:38:28.315 INFO:tasks.ceph.mon.a.smithi052.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55e374db9a71] 2021-03-09T09:38:28.315 INFO:tasks.ceph.mon.a.smithi052.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55e374de9295] 2021-03-09T09:38:28.315 INFO:tasks.ceph.mon.a.smithi052.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55e374df124b] 2021-03-09T09:38:28.316 INFO:tasks.ceph.mon.a.smithi052.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55e374df17ab] 2021-03-09T09:38:28.316 INFO:tasks.ceph.mon.a.smithi052.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55e374d60ade] 2021-03-09T09:38:28.316 INFO:tasks.ceph.mon.a.smithi052.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55e374c4bc79] 2021-03-09T09:38:28.316 INFO:tasks.ceph.mon.a.smithi052.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55e374c4f229] 2021-03-09T09:38:28.317 INFO:tasks.ceph.mon.a.smithi052.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.318 INFO:tasks.ceph.mon.a.smithi052.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55e374c5187f] 2021-03-09T09:38:28.318 INFO:tasks.ceph.mon.a.smithi052.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55e374c4fd70] 2021-03-09T09:38:28.318 INFO:tasks.ceph.mon.a.smithi052.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.318 INFO:tasks.ceph.mon.a.smithi052.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55e374c7e178] 2021-03-09T09:38:28.318 INFO:tasks.ceph.mon.a.smithi052.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7fc5abcf42ca] 2021-03-09T09:38:28.319 INFO:tasks.ceph.mon.a.smithi052.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fc5abd9e71d] 2021-03-09T09:38:28.319 INFO:tasks.ceph.mon.a.smithi052.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc5aa9626db] 2021-03-09T09:38:28.319 INFO:tasks.ceph.mon.a.smithi052.stderr: 22: clone() 2021-03-09T09:38:28.319 INFO:tasks.ceph.mon.a.smithi052.stderr:2021-03-09T09:38:28.307+0000 7fc59f330700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:38:28.320 INFO:tasks.ceph.mon.a.smithi052.stderr: in thread 7fc59f330700 thread_name:ms_dispatch 2021-03-09T09:38:28.320 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.320 INFO:tasks.ceph.mon.a.smithi052.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:28.320 INFO:tasks.ceph.mon.a.smithi052.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fc5aa96d980] 2021-03-09T09:38:28.320 INFO:tasks.ceph.mon.a.smithi052.stderr: 2: gsignal() 2021-03-09T09:38:28.321 INFO:tasks.ceph.mon.a.smithi052.stderr: 3: abort() 2021-03-09T09:38:28.321 INFO:tasks.ceph.mon.a.smithi052.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fc5abaf242e] 2021-03-09T09:38:28.321 INFO:tasks.ceph.mon.a.smithi052.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fc5abaf25b8] 2021-03-09T09:38:28.321 INFO:tasks.ceph.mon.a.smithi052.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55e374d6da65] 2021-03-09T09:38:28.321 INFO:tasks.ceph.mon.a.smithi052.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55e374db9a71] 2021-03-09T09:38:28.322 INFO:tasks.ceph.mon.a.smithi052.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55e374de9295] 2021-03-09T09:38:28.322 INFO:tasks.ceph.mon.a.smithi052.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55e374df124b] 2021-03-09T09:38:28.322 INFO:tasks.ceph.mon.a.smithi052.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55e374df17ab] 2021-03-09T09:38:28.322 INFO:tasks.ceph.mon.a.smithi052.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55e374d60ade] 2021-03-09T09:38:28.323 INFO:tasks.ceph.mon.a.smithi052.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55e374c4bc79] 2021-03-09T09:38:28.323 INFO:tasks.ceph.mon.a.smithi052.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55e374c4f229] 2021-03-09T09:38:28.323 INFO:tasks.ceph.mon.a.smithi052.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.323 INFO:tasks.ceph.mon.a.smithi052.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55e374c5187f] 2021-03-09T09:38:28.323 INFO:tasks.ceph.mon.a.smithi052.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55e374c4fd70] 2021-03-09T09:38:28.324 INFO:tasks.ceph.mon.a.smithi052.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.324 INFO:tasks.ceph.mon.a.smithi052.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55e374c7e178] 2021-03-09T09:38:28.324 INFO:tasks.ceph.mon.a.smithi052.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7fc5abcf42ca] 2021-03-09T09:38:28.324 INFO:tasks.ceph.mon.a.smithi052.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fc5abd9e71d] 2021-03-09T09:38:28.325 INFO:tasks.ceph.mon.a.smithi052.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc5aa9626db] 2021-03-09T09:38:28.325 INFO:tasks.ceph.mon.a.smithi052.stderr: 22: clone() 2021-03-09T09:38:28.325 INFO:tasks.ceph.mon.a.smithi052.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:38:28.325 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.326 INFO:tasks.ceph.mon.a.smithi052.stderr: -1> 2021-03-09T09:38:28.299+0000 7fc59f330700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc59f330700 time 2021-03-09T09:38:28.290285+0000 2021-03-09T09:38:28.326 INFO:tasks.ceph.mon.a.smithi052.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:38:28.326 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.327 INFO:tasks.ceph.mon.a.smithi052.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:28.327 INFO:tasks.ceph.mon.a.smithi052.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fc5abaf23dd] 2021-03-09T09:38:28.327 INFO:tasks.ceph.mon.a.smithi052.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fc5abaf25b8] 2021-03-09T09:38:28.327 INFO:tasks.ceph.mon.a.smithi052.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55e374d6da65] 2021-03-09T09:38:28.328 INFO:tasks.ceph.mon.a.smithi052.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55e374db9a71] 2021-03-09T09:38:28.328 INFO:tasks.ceph.mon.a.smithi052.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55e374de9295] 2021-03-09T09:38:28.328 INFO:tasks.ceph.mon.a.smithi052.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55e374df124b] 2021-03-09T09:38:28.328 INFO:tasks.ceph.mon.a.smithi052.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55e374df17ab] 2021-03-09T09:38:28.329 INFO:tasks.ceph.mon.a.smithi052.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55e374d60ade] 2021-03-09T09:38:28.329 INFO:tasks.ceph.mon.a.smithi052.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55e374c4bc79] 2021-03-09T09:38:28.329 INFO:tasks.ceph.mon.a.smithi052.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55e374c4f229] 2021-03-09T09:38:28.329 INFO:tasks.ceph.mon.a.smithi052.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.329 INFO:tasks.ceph.mon.a.smithi052.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55e374c5187f] 2021-03-09T09:38:28.330 INFO:tasks.ceph.mon.a.smithi052.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55e374c4fd70] 2021-03-09T09:38:28.330 INFO:tasks.ceph.mon.a.smithi052.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.330 INFO:tasks.ceph.mon.a.smithi052.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55e374c7e178] 2021-03-09T09:38:28.330 INFO:tasks.ceph.mon.a.smithi052.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7fc5abcf42ca] 2021-03-09T09:38:28.331 INFO:tasks.ceph.mon.a.smithi052.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fc5abd9e71d] 2021-03-09T09:38:28.331 INFO:tasks.ceph.mon.a.smithi052.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc5aa9626db] 2021-03-09T09:38:28.331 INFO:tasks.ceph.mon.a.smithi052.stderr: 19: clone() 2021-03-09T09:38:28.331 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.331 INFO:tasks.ceph.mon.a.smithi052.stderr: 0> 2021-03-09T09:38:28.307+0000 7fc59f330700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:38:28.332 INFO:tasks.ceph.mon.a.smithi052.stderr: in thread 7fc59f330700 thread_name:ms_dispatch 2021-03-09T09:38:28.332 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.332 INFO:tasks.ceph.mon.a.smithi052.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:28.332 INFO:tasks.ceph.mon.a.smithi052.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fc5aa96d980] 2021-03-09T09:38:28.333 INFO:tasks.ceph.mon.a.smithi052.stderr: 2: gsignal() 2021-03-09T09:38:28.333 INFO:tasks.ceph.mon.a.smithi052.stderr: 3: abort() 2021-03-09T09:38:28.333 INFO:tasks.ceph.mon.a.smithi052.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fc5abaf242e] 2021-03-09T09:38:28.333 INFO:tasks.ceph.mon.a.smithi052.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fc5abaf25b8] 2021-03-09T09:38:28.333 INFO:tasks.ceph.mon.a.smithi052.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55e374d6da65] 2021-03-09T09:38:28.334 INFO:tasks.ceph.mon.a.smithi052.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55e374db9a71] 2021-03-09T09:38:28.334 INFO:tasks.ceph.mon.a.smithi052.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55e374de9295] 2021-03-09T09:38:28.334 INFO:tasks.ceph.mon.a.smithi052.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55e374df124b] 2021-03-09T09:38:28.334 INFO:tasks.ceph.mon.a.smithi052.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55e374df17ab] 2021-03-09T09:38:28.335 INFO:tasks.ceph.mon.a.smithi052.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55e374d60ade] 2021-03-09T09:38:28.335 INFO:tasks.ceph.mon.a.smithi052.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55e374c4bc79] 2021-03-09T09:38:28.335 INFO:tasks.ceph.mon.a.smithi052.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55e374c4f229] 2021-03-09T09:38:28.335 INFO:tasks.ceph.mon.a.smithi052.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.335 INFO:tasks.ceph.mon.a.smithi052.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55e374c5187f] 2021-03-09T09:38:28.336 INFO:tasks.ceph.mon.a.smithi052.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55e374c4fd70] 2021-03-09T09:38:28.336 INFO:tasks.ceph.mon.a.smithi052.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.336 INFO:tasks.ceph.mon.a.smithi052.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55e374c7e178] 2021-03-09T09:38:28.336 INFO:tasks.ceph.mon.a.smithi052.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7fc5abcf42ca] 2021-03-09T09:38:28.337 INFO:tasks.ceph.mon.a.smithi052.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fc5abd9e71d] 2021-03-09T09:38:28.337 INFO:tasks.ceph.mon.a.smithi052.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc5aa9626db] 2021-03-09T09:38:28.337 INFO:tasks.ceph.mon.a.smithi052.stderr: 22: clone() 2021-03-09T09:38:28.337 INFO:tasks.ceph.mon.a.smithi052.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:38:28.337 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.338 INFO:tasks.ceph.mon.a.smithi052.stderr: -1> 2021-03-09T09:38:28.299+0000 7fc59f330700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc59f330700 time 2021-03-09T09:38:28.290285+0000 2021-03-09T09:38:28.338 INFO:tasks.ceph.mon.a.smithi052.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:38:28.339 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.339 INFO:tasks.ceph.mon.a.smithi052.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:28.339 INFO:tasks.ceph.mon.a.smithi052.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fc5abaf23dd] 2021-03-09T09:38:28.339 INFO:tasks.ceph.mon.a.smithi052.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fc5abaf25b8] 2021-03-09T09:38:28.340 INFO:tasks.ceph.mon.a.smithi052.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55e374d6da65] 2021-03-09T09:38:28.340 INFO:tasks.ceph.mon.a.smithi052.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55e374db9a71] 2021-03-09T09:38:28.340 INFO:tasks.ceph.mon.a.smithi052.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55e374de9295] 2021-03-09T09:38:28.340 INFO:tasks.ceph.mon.a.smithi052.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55e374df124b] 2021-03-09T09:38:28.340 INFO:tasks.ceph.mon.a.smithi052.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55e374df17ab] 2021-03-09T09:38:28.341 INFO:tasks.ceph.mon.a.smithi052.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55e374d60ade] 2021-03-09T09:38:28.341 INFO:tasks.ceph.mon.a.smithi052.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55e374c4bc79] 2021-03-09T09:38:28.341 INFO:tasks.ceph.mon.a.smithi052.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55e374c4f229] 2021-03-09T09:38:28.341 INFO:tasks.ceph.mon.a.smithi052.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.342 INFO:tasks.ceph.mon.a.smithi052.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55e374c5187f] 2021-03-09T09:38:28.342 INFO:tasks.ceph.mon.a.smithi052.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55e374c4fd70] 2021-03-09T09:38:28.342 INFO:tasks.ceph.mon.a.smithi052.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.342 INFO:tasks.ceph.mon.a.smithi052.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55e374c7e178] 2021-03-09T09:38:28.343 INFO:tasks.ceph.mon.a.smithi052.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7fc5abcf42ca] 2021-03-09T09:38:28.343 INFO:tasks.ceph.mon.a.smithi052.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fc5abd9e71d] 2021-03-09T09:38:28.343 INFO:tasks.ceph.mon.a.smithi052.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc5aa9626db] 2021-03-09T09:38:28.343 INFO:tasks.ceph.mon.a.smithi052.stderr: 19: clone() 2021-03-09T09:38:28.344 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.344 INFO:tasks.ceph.mon.a.smithi052.stderr: 0> 2021-03-09T09:38:28.307+0000 7fc59f330700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:38:28.344 INFO:tasks.ceph.mon.a.smithi052.stderr: in thread 7fc59f330700 thread_name:ms_dispatch 2021-03-09T09:38:28.344 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.344 INFO:tasks.ceph.mon.a.smithi052.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:28.345 INFO:tasks.ceph.mon.a.smithi052.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fc5aa96d980] 2021-03-09T09:38:28.345 INFO:tasks.ceph.mon.a.smithi052.stderr: 2: gsignal() 2021-03-09T09:38:28.345 INFO:tasks.ceph.mon.a.smithi052.stderr: 3: abort() 2021-03-09T09:38:28.345 INFO:tasks.ceph.mon.a.smithi052.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fc5abaf242e] 2021-03-09T09:38:28.346 INFO:tasks.ceph.mon.a.smithi052.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fc5abaf25b8] 2021-03-09T09:38:28.346 INFO:tasks.ceph.mon.a.smithi052.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55e374d6da65] 2021-03-09T09:38:28.346 INFO:tasks.ceph.mon.a.smithi052.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55e374db9a71] 2021-03-09T09:38:28.346 INFO:tasks.ceph.mon.a.smithi052.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55e374de9295] 2021-03-09T09:38:28.346 INFO:tasks.ceph.mon.a.smithi052.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55e374df124b] 2021-03-09T09:38:28.347 INFO:tasks.ceph.mon.a.smithi052.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55e374df17ab] 2021-03-09T09:38:28.347 INFO:tasks.ceph.mon.a.smithi052.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55e374d60ade] 2021-03-09T09:38:28.347 INFO:tasks.ceph.mon.a.smithi052.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55e374c4bc79] 2021-03-09T09:38:28.347 INFO:tasks.ceph.mon.a.smithi052.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55e374c4f229] 2021-03-09T09:38:28.348 INFO:tasks.ceph.mon.a.smithi052.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.348 INFO:tasks.ceph.mon.a.smithi052.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55e374c5187f] 2021-03-09T09:38:28.348 INFO:tasks.ceph.mon.a.smithi052.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55e374c4fd70] 2021-03-09T09:38:28.348 INFO:tasks.ceph.mon.a.smithi052.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55e374c5066a] 2021-03-09T09:38:28.348 INFO:tasks.ceph.mon.a.smithi052.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55e374c7e178] 2021-03-09T09:38:28.349 INFO:tasks.ceph.mon.a.smithi052.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7fc5abcf42ca] 2021-03-09T09:38:28.349 INFO:tasks.ceph.mon.a.smithi052.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fc5abd9e71d] 2021-03-09T09:38:28.349 INFO:tasks.ceph.mon.a.smithi052.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc5aa9626db] 2021-03-09T09:38:28.349 INFO:tasks.ceph.mon.a.smithi052.stderr: 22: clone() 2021-03-09T09:38:28.349 INFO:tasks.ceph.mon.a.smithi052.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:38:28.350 INFO:tasks.ceph.mon.a.smithi052.stderr: 2021-03-09T09:38:28.795 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T09:38:28.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f556cf09700 1 Processor -- start 2021-03-09T09:38:28.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f556cf09700 1 -- start start 2021-03-09T09:38:28.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f556cf09700 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f556809e1a0 0x7f55680adb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:28.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f556cf09700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f5568136060 0x7f5568136490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:28.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f556cf09700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556809e580 con 0x7f556809f2d0 2021-03-09T09:38:28.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f556cf09700 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556809f7a0 con 0x7f556809e1a0 2021-03-09T09:38:28.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f556cf09700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556809fa40 con 0x7f5568136060 2021-03-09T09:38:28.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f556659c700 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f556809e1a0 msgr2=0x7f55680adb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:38:28.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f556659c700 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f556809e1a0 0x7f55680adb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:28.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f5566d9d700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f5568136060 0x7f5568136490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:28.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f5566d9d700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f5568136060 0x7f5568136490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.52:57002/0 (socket says 172.21.15.52:57002) 2021-03-09T09:38:28.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f5565d9b700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f556809f2d0 0x7f55680a9c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.52:60592/0 (socket says 172.21.15.52:60592) 2021-03-09T09:38:28.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.971+0000 7f5566d9d700 1 -- 172.21.15.52:0/3013515789 learned_addr learned my addr 172.21.15.52:0/3013515789 (peer_addr_for_me v2:172.21.15.52:0/0) 2021-03-09T09:38:28.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f5566d9d700 1 -- 172.21.15.52:0/3013515789 >> v2:172.21.15.52:3300/0 conn(0x7f556809e1a0 msgr2=0x7f55680adb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T09:38:28.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f5566d9d700 1 --2- 172.21.15.52:0/3013515789 >> v2:172.21.15.52:3300/0 conn(0x7f556809e1a0 0x7f55680adb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:28.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f5566d9d700 1 -- 172.21.15.52:0/3013515789 >> v1:172.21.15.194:6789/0 conn(0x7f556809f2d0 legacy=0x7f55680a9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:28.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556559a700 1 -- 172.21.15.52:0/3013515789 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059177480 0 0) 0x7f556809e580 con 0x7f556809f2d0 2021-03-09T09:38:28.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f5566d9d700 1 -- 172.21.15.52:0/3013515789 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55680ae040 con 0x7f5568136060 2021-03-09T09:38:28.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f5566d9d700 1 --2- 172.21.15.52:0/3013515789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5568136060 0x7f5568136490 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f556809d410 tx=0x7f5560009290).ready entity=mon.1 client_cookie=be4578fb8d00bd35 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:28.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556559a700 1 -- 172.21.15.52:0/3013515789 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556000bbc0 con 0x7f5568136060 2021-03-09T09:38:28.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556559a700 1 -- 172.21.15.52:0/3013515789 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556000bd20 con 0x7f5568136060 2021-03-09T09:38:28.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556559a700 1 -- 172.21.15.52:0/3013515789 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5560011610 con 0x7f5568136060 2021-03-09T09:38:28.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556cf09700 1 -- 172.21.15.52:0/3013515789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5568136060 msgr2=0x7f5568136490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:28.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556cf09700 1 --2- 172.21.15.52:0/3013515789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5568136060 0x7f5568136490 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f556809d410 tx=0x7f5560009290).stop 2021-03-09T09:38:28.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556cf09700 1 -- 172.21.15.52:0/3013515789 shutdown_connections 2021-03-09T09:38:28.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556cf09700 1 --2- 172.21.15.52:0/3013515789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5568136060 0x7f5568136490 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:28.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556cf09700 1 --2- 172.21.15.52:0/3013515789 >> v2:172.21.15.52:3300/0 conn(0x7f556809e1a0 0x7f55680adb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:28.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556cf09700 1 -- 172.21.15.52:0/3013515789 >> 172.21.15.52:0/3013515789 conn(0x7f5568065f70 msgr2=0x7f5568066380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:28.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556cf09700 1 -- 172.21.15.52:0/3013515789 shutdown_connections 2021-03-09T09:38:28.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556cf09700 1 -- 172.21.15.52:0/3013515789 wait complete. 2021-03-09T09:38:28.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556cf09700 1 Processor -- start 2021-03-09T09:38:28.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.975+0000 7f556cf09700 1 -- start start 2021-03-09T09:38:28.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556cf09700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f556809e1a0 0x7f55680a8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:28.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556cf09700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f556809f2d0 0x7f5568062910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:28.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556cf09700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5568136060 0x7f55680a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:28.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556cf09700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5568139ef0 con 0x7f556809f2d0 2021-03-09T09:38:28.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556cf09700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556813a120 con 0x7f5568136060 2021-03-09T09:38:28.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556cf09700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556813a3c0 con 0x7f556809e1a0 2021-03-09T09:38:28.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f5565d9b700 1 -- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f556809f2d0 msgr2=0x7f5568062910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:38:28.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f5565d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f556809f2d0 0x7f5568062910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:28.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556659c700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f556809e1a0 0x7f55680a8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:28.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556659c700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f556809e1a0 0x7f55680a8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3301/0 says I am v2:172.21.15.52:35126/0 (socket says 172.21.15.52:35126) 2021-03-09T09:38:28.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556659c700 1 -- 172.21.15.52:0/498878863 learned_addr learned my addr 172.21.15.52:0/498878863 (peer_addr_for_me v2:172.21.15.52:0/0) 2021-03-09T09:38:28.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f5566d9d700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5568136060 0x7f55680a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:29.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556659c700 1 -- 172.21.15.52:0/498878863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5568136060 msgr2=0x7f55680a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:29.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556659c700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5568136060 0x7f55680a5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:29.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556659c700 1 -- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f556809f2d0 msgr2=0x7f5568062910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:29.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556659c700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f556809f2d0 0x7f5568062910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:29.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556659c700 1 -- 172.21.15.52:0/498878863 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556813ba60 con 0x7f556809e1a0 2021-03-09T09:38:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556659c700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f556809e1a0 0x7f55680a8740 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f555c009730 tx=0x7f555c006e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f55567fc700 1 -- 172.21.15.52:0/498878863 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c006170 con 0x7f556809e1a0 2021-03-09T09:38:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f55567fc700 1 -- 172.21.15.52:0/498878863 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c0062d0 con 0x7f556809e1a0 2021-03-09T09:38:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556cf09700 1 -- 172.21.15.52:0/498878863 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5568061320 con 0x7f556809e1a0 2021-03-09T09:38:29.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f55567fc700 1 -- 172.21.15.52:0/498878863 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c0038d0 con 0x7f556809e1a0 2021-03-09T09:38:29.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556cf09700 1 -- 172.21.15.52:0/498878863 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55680a6330 con 0x7f556809e1a0 2021-03-09T09:38:29.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.979+0000 7f556cf09700 1 -- 172.21.15.52:0/498878863 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55680a5a90 con 0x7f556809e1a0 2021-03-09T09:38:29.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.983+0000 7f55567fc700 1 -- 172.21.15.52:0/498878863 <== mon.2 v2:172.21.15.52:3301/0 4 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f555c00f070 con 0x7f556809e1a0 2021-03-09T09:38:29.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.983+0000 7f55567fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:29.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.983+0000 7f55567fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:29.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.983+0000 7f55567fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:38:29.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.983+0000 7f55567fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:38:29.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.983+0000 7f55567fc700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7f5544083100 0x7f55440855c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:29.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.983+0000 7f55567fc700 1 -- 172.21.15.52:0/498878863 <== mon.2 v2:172.21.15.52:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 2212+0+0 (secure 0 0 0) 0x7f555c0a4500 con 0x7f556809e1a0 2021-03-09T09:38:29.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.983+0000 7f5565d9b700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7f5544083100 0x7f55440855c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:29.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.983+0000 7f5565d9b700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7f5544083100 0x7f55440855c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f55580010a0 tx=0x7f5558007770).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:29.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:28.987+0000 7f55567fc700 1 -- 172.21.15.52:0/498878863 <== mon.2 v2:172.21.15.52:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+149001 (secure 0 0 0) 0x7f555c0795d0 con 0x7f556809e1a0 2021-03-09T09:38:29.111 INFO:tasks.ceph.mon.a.smithi052.stderr:daemon-helper: command crashed with signal 6 2021-03-09T09:38:29.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.143+0000 7f556cf09700 1 -- 172.21.15.52:0/498878863 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f556805e210 con 0x7f556809e1a0 2021-03-09T09:38:29.147 INFO:teuthology.misc.health.smithi052.stdout: 2021-03-09T09:38:29.147 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":9,"fsid":"1325f083-8043-46e9-93ad-0afef33c673d","created":"2021-03-09T09:38:19.242162+0000","modified":"2021-03-09T09:38:28.207203+0000","last_up_change":"2021-03-09T09:38:28.207203+0000","last_in_change":"2021-03-09T09:38:21.688382+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"76d971ae-5519-47c8-a3d6-668100f48891","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":14575},{"type":"v1","addr":"172.21.15.52:6801","nonce":14575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":14575},{"type":"v1","addr":"172.21.15.52:6803","nonce":14575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":14575},{"type":"v1","addr":"172.21.15.52:6807","nonce":14575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":14575},{"type":"v1","addr":"172.21.15.52:6805","nonce":14575}]},"public_addr":"172.21.15.52:6801/14575","cluster_addr":"172.21.15.52:6803/14575","heartbeat_back_addr":"172.21.15.52:6807/14575","heartbeat_front_addr":"172.21.15.52:6805/14575","state":["exists","up"]},{"osd":1,"uuid":"fe38e427-1abd-4b6c-a593-a5dc62b99540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":16572},{"type":"v1","addr":"172.21.15.194:6809","nonce":16572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":16572},{"type":"v1","addr":"172.21.15.194:6811","nonce":16572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":16572},{"type":"v1","addr":"172.21.15.194:6815","nonce":16572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":16572},{"type":"v1","addr":"172.21.15.194:6813","nonce":16572}]},"public_addr":"172.21.15.194:6809/16572","cluster_addr":"172.21.15.194:6811/16572","heartbeat_back_addr":"172.21.15.194:6815/16572","heartbeat_front_addr":"172.21.15.194:6813/16572","state":["exists","up"]},{"osd":2,"uuid":"63b3755a-f879-4c75-b2dd-eeb09d99f7ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b9634dd7-a1fe-4bbd-950a-af1aba0e19fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-09T09:38:29.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.143+0000 7f55567fc700 1 -- 172.21.15.52:0/498878863 <== mon.2 v2:172.21.15.52:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v9) v1 ==== 74+0+4979 (secure 0 0 0) 0x7f555c018030 con 0x7f556809e1a0 2021-03-09T09:38:29.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 10 mgrc shutdown 2021-03-09T09:38:29.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 -- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7f5544083100 msgr2=0x7f55440855c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:29.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7f5544083100 0x7f55440855c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f55580010a0 tx=0x7f5558007770).stop 2021-03-09T09:38:29.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 -- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f556809e1a0 msgr2=0x7f55680a8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:29.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f556809e1a0 0x7f55680a8740 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f555c009730 tx=0x7f555c006e10).stop 2021-03-09T09:38:29.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 -- 172.21.15.52:0/498878863 shutdown_connections 2021-03-09T09:38:29.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5568136060 0x7f55680a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:29.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7f5544083100 0x7f55440855c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:29.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7f556809e1a0 0x7f55680a8740 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:29.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 --2- 172.21.15.52:0/498878863 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f556809f2d0 0x7f5568062910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:29.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 -- 172.21.15.52:0/498878863 >> 172.21.15.52:0/498878863 conn(0x7f5568065f70 msgr2=0x7f556813cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:29.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 -- 172.21.15.52:0/498878863 shutdown_connections 2021-03-09T09:38:29.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:29.147+0000 7f556cf09700 1 -- 172.21.15.52:0/498878863 wait complete. 2021-03-09T09:38:29.174 DEBUG:teuthology.misc:2 of 4 OSDs are up 2021-03-09T09:38:35.579 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T09:38:35.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb4cddc700 1 Processor -- start 2021-03-09T09:38:35.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb4cddc700 1 -- start start 2021-03-09T09:38:35.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb4cddc700 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fcb4805ca90 0x7fcb480a0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:35.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb4cddc700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480aca50 0x7fcb480672d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:35.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb4cddc700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480a0f80 con 0x7fcb480a0a30 2021-03-09T09:38:35.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb4cddc700 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4813c550 con 0x7fcb4805ca90 2021-03-09T09:38:35.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb4cddc700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480a17a0 con 0x7fcb480aca50 2021-03-09T09:38:35.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb477fe700 1 -- >> v2:172.21.15.52:3300/0 conn(0x7fcb4805ca90 msgr2=0x7fcb480a0540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:38:35.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb477fe700 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fcb4805ca90 0x7fcb480a0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:35.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb46ffd700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480aca50 0x7fcb480672d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:35.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb46ffd700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480aca50 0x7fcb480672d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3301/0 says I am v2:172.21.15.52:35144/0 (socket says 172.21.15.52:35144) 2021-03-09T09:38:35.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb46ffd700 1 -- 172.21.15.52:0/2426272564 learned_addr learned my addr 172.21.15.52:0/2426272564 (peer_addr_for_me v2:172.21.15.52:0/0) 2021-03-09T09:38:35.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb46ffd700 1 -- 172.21.15.52:0/2426272564 >> v2:172.21.15.52:3300/0 conn(0x7fcb4805ca90 msgr2=0x7fcb480a0540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T09:38:35.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb46ffd700 1 --2- 172.21.15.52:0/2426272564 >> v2:172.21.15.52:3300/0 conn(0x7fcb4805ca90 0x7fcb480a0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:35.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb46ffd700 1 -- 172.21.15.52:0/2426272564 >> v1:172.21.15.194:6789/0 conn(0x7fcb480a0a30 legacy=0x7fcb480677c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:35.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb46ffd700 1 -- 172.21.15.52:0/2426272564 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4805ce70 con 0x7fcb480aca50 2021-03-09T09:38:35.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb46ffd700 1 --2- 172.21.15.52:0/2426272564 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480aca50 0x7fcb480672d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fcb3c00b9e0 tx=0x7fcb3c006ea0).ready entity=mon.2 client_cookie=d95ee8ef66a55330 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:35.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb467fc700 1 -- 172.21.15.52:0/2426272564 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c0098d0 con 0x7fcb480aca50 2021-03-09T09:38:35.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb467fc700 1 -- 172.21.15.52:0/2426272564 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c009a30 con 0x7fcb480aca50 2021-03-09T09:38:35.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb467fc700 1 -- 172.21.15.52:0/2426272564 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c009db0 con 0x7fcb480aca50 2021-03-09T09:38:35.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb4cddc700 1 -- 172.21.15.52:0/2426272564 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480aca50 msgr2=0x7fcb480672d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:35.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.759+0000 7fcb4cddc700 1 --2- 172.21.15.52:0/2426272564 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480aca50 0x7fcb480672d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fcb3c00b9e0 tx=0x7fcb3c006ea0).stop 2021-03-09T09:38:35.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 -- 172.21.15.52:0/2426272564 shutdown_connections 2021-03-09T09:38:35.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 --2- 172.21.15.52:0/2426272564 >> v2:172.21.15.52:3300/0 conn(0x7fcb4805ca90 0x7fcb480a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:35.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 --2- 172.21.15.52:0/2426272564 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480aca50 0x7fcb480672d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:35.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 -- 172.21.15.52:0/2426272564 >> 172.21.15.52:0/2426272564 conn(0x7fcb48061890 msgr2=0x7fcb480631c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:35.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 -- 172.21.15.52:0/2426272564 shutdown_connections 2021-03-09T09:38:35.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 -- 172.21.15.52:0/2426272564 wait complete. 2021-03-09T09:38:35.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 Processor -- start 2021-03-09T09:38:35.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 -- start start 2021-03-09T09:38:35.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb4805ca90 0x7fcb480a8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:35.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480a0a30 0x7fcb480aaf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:35.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb480aca50 0x7fcb480ab880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:35.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480ab5e0 con 0x7fcb480aca50 2021-03-09T09:38:35.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480a7a90 con 0x7fcb4805ca90 2021-03-09T09:38:35.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb4cddc700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480a8460 con 0x7fcb480a0a30 2021-03-09T09:38:35.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb47fff700 1 -- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb480aca50 msgr2=0x7fcb480ab880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:38:35.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb47fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb480aca50 0x7fcb480ab880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:35.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb46ffd700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480a0a30 0x7fcb480aaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:35.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb46ffd700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480a0a30 0x7fcb480aaf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3301/0 says I am v2:172.21.15.52:35150/0 (socket says 172.21.15.52:35150) 2021-03-09T09:38:35.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb46ffd700 1 -- 172.21.15.52:0/306082637 learned_addr learned my addr 172.21.15.52:0/306082637 (peer_addr_for_me v2:172.21.15.52:0/0) 2021-03-09T09:38:35.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb477fe700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb4805ca90 0x7fcb480a8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:35.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb46ffd700 1 -- 172.21.15.52:0/306082637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb4805ca90 msgr2=0x7fcb480a8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:35.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb46ffd700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb4805ca90 0x7fcb480a8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:35.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb46ffd700 1 -- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb480aca50 msgr2=0x7fcb480ab880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:35.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb46ffd700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb480aca50 0x7fcb480ab880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:35.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb46ffd700 1 -- 172.21.15.52:0/306082637 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4813f030 con 0x7fcb480a0a30 2021-03-09T09:38:35.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb46ffd700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480a0a30 0x7fcb480aaf20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fcb3c006ee0 tx=0x7fcb3c016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:35.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.763+0000 7fcb2f7fe700 1 -- 172.21.15.52:0/306082637 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c01c420 con 0x7fcb480a0a30 2021-03-09T09:38:35.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb2f7fe700 1 -- 172.21.15.52:0/306082637 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c01c580 con 0x7fcb480a0a30 2021-03-09T09:38:35.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb4cddc700 1 -- 172.21.15.52:0/306082637 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb48148110 con 0x7fcb480a0a30 2021-03-09T09:38:35.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb2f7fe700 1 -- 172.21.15.52:0/306082637 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c01c940 con 0x7fcb480a0a30 2021-03-09T09:38:35.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb4cddc700 1 -- 172.21.15.52:0/306082637 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb480abf90 con 0x7fcb480a0a30 2021-03-09T09:38:35.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb4cddc700 1 -- 172.21.15.52:0/306082637 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb480a8620 con 0x7fcb480a0a30 2021-03-09T09:38:35.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb2f7fe700 1 -- 172.21.15.52:0/306082637 <== mon.2 v2:172.21.15.52:3301/0 4 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7fcb3c00f070 con 0x7fcb480a0a30 2021-03-09T09:38:35.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb2f7fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:35.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb2f7fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:35.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb2f7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:38:35.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb2f7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:38:35.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.767+0000 7fcb2f7fe700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7fcb28083130 0x7fcb280855f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:35.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.771+0000 7fcb2f7fe700 1 -- 172.21.15.52:0/306082637 <== mon.2 v2:172.21.15.52:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 2212+0+0 (secure 0 0 0) 0x7fcb3c0a60e0 con 0x7fcb480a0a30 2021-03-09T09:38:35.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.771+0000 7fcb477fe700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7fcb28083130 0x7fcb280855f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:35.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.771+0000 7fcb477fe700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7fcb28083130 0x7fcb280855f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fcb38003690 tx=0x7fcb38006c40).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:35.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.779+0000 7fcb2f7fe700 1 -- 172.21.15.52:0/306082637 <== mon.2 v2:172.21.15.52:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+149001 (secure 0 0 0) 0x7fcb3c07b2c0 con 0x7fcb480a0a30 2021-03-09T09:38:35.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:35.935+0000 7fcb4cddc700 1 -- 172.21.15.52:0/306082637 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb480589b0 con 0x7fcb480a0a30 2021-03-09T09:38:43.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.247+0000 7fcb2f7fe700 1 -- 172.21.15.52:0/306082637 <== mon.2 v2:172.21.15.52:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v9) v1 ==== 74+0+4979 (secure 0 0 0) 0x7fcb3c07e8e0 con 0x7fcb480a0a30 2021-03-09T09:38:43.252 INFO:teuthology.misc.health.smithi052.stdout: 2021-03-09T09:38:43.253 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":9,"fsid":"1325f083-8043-46e9-93ad-0afef33c673d","created":"2021-03-09T09:38:19.242162+0000","modified":"2021-03-09T09:38:28.207203+0000","last_up_change":"2021-03-09T09:38:28.207203+0000","last_in_change":"2021-03-09T09:38:21.688382+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"76d971ae-5519-47c8-a3d6-668100f48891","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":14575},{"type":"v1","addr":"172.21.15.52:6801","nonce":14575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":14575},{"type":"v1","addr":"172.21.15.52:6803","nonce":14575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":14575},{"type":"v1","addr":"172.21.15.52:6807","nonce":14575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":14575},{"type":"v1","addr":"172.21.15.52:6805","nonce":14575}]},"public_addr":"172.21.15.52:6801/14575","cluster_addr":"172.21.15.52:6803/14575","heartbeat_back_addr":"172.21.15.52:6807/14575","heartbeat_front_addr":"172.21.15.52:6805/14575","state":["exists","up"]},{"osd":1,"uuid":"fe38e427-1abd-4b6c-a593-a5dc62b99540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":16572},{"type":"v1","addr":"172.21.15.194:6809","nonce":16572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":16572},{"type":"v1","addr":"172.21.15.194:6811","nonce":16572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":16572},{"type":"v1","addr":"172.21.15.194:6815","nonce":16572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":16572},{"type":"v1","addr":"172.21.15.194:6813","nonce":16572}]},"public_addr":"172.21.15.194:6809/16572","cluster_addr":"172.21.15.194:6811/16572","heartbeat_back_addr":"172.21.15.194:6815/16572","heartbeat_front_addr":"172.21.15.194:6813/16572","state":["exists","up"]},{"osd":2,"uuid":"63b3755a-f879-4c75-b2dd-eeb09d99f7ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b9634dd7-a1fe-4bbd-950a-af1aba0e19fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-09T09:38:43.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 10 mgrc shutdown 2021-03-09T09:38:43.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 -- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7fcb28083130 msgr2=0x7fcb280855f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:43.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7fcb28083130 0x7fcb280855f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fcb38003690 tx=0x7fcb38006c40).stop 2021-03-09T09:38:43.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 -- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480a0a30 msgr2=0x7fcb480aaf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:43.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480a0a30 0x7fcb480aaf20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fcb3c006ee0 tx=0x7fcb3c016000).stop 2021-03-09T09:38:43.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 -- 172.21.15.52:0/306082637 shutdown_connections 2021-03-09T09:38:43.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb480aca50 0x7fcb480ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:43.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb4805ca90 0x7fcb480a8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:43.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7fcb28083130 0x7fcb280855f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:43.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 --2- 172.21.15.52:0/306082637 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7fcb480a0a30 0x7fcb480aaf20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:43.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 -- 172.21.15.52:0/306082637 >> 172.21.15.52:0/306082637 conn(0x7fcb48061890 msgr2=0x7fcb4809c550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:43.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 -- 172.21.15.52:0/306082637 shutdown_connections 2021-03-09T09:38:43.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:43.255+0000 7fcb4cddc700 1 -- 172.21.15.52:0/306082637 wait complete. 2021-03-09T09:38:43.280 DEBUG:teuthology.misc:2 of 4 OSDs are up 2021-03-09T09:38:43.331 INFO:tasks.ceph.mon.b.smithi194.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f5e111f0700 time 2021-03-09T09:38:43.333866+0000 2021-03-09T09:38:43.332 INFO:tasks.ceph.mon.b.smithi194.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:38:43.347 INFO:tasks.ceph.mon.b.smithi194.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:43.348 INFO:tasks.ceph.mon.b.smithi194.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5e1d9b23dd] 2021-03-09T09:38:43.348 INFO:tasks.ceph.mon.b.smithi194.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5e1d9b25b8] 2021-03-09T09:38:43.348 INFO:tasks.ceph.mon.b.smithi194.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x564b40fc7a65] 2021-03-09T09:38:43.348 INFO:tasks.ceph.mon.b.smithi194.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x564b41013a71] 2021-03-09T09:38:43.349 INFO:tasks.ceph.mon.b.smithi194.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x564b41043295] 2021-03-09T09:38:43.349 INFO:tasks.ceph.mon.b.smithi194.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x564b4104b24b] 2021-03-09T09:38:43.349 INFO:tasks.ceph.mon.b.smithi194.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x564b4104b7ab] 2021-03-09T09:38:43.349 INFO:tasks.ceph.mon.b.smithi194.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x564b40fbaade] 2021-03-09T09:38:43.350 INFO:tasks.ceph.mon.b.smithi194.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x564b40f10134] 2021-03-09T09:38:43.350 INFO:tasks.ceph.mon.b.smithi194.stderr: 10: (C_MonOp::finish(int)+0x45) [0x564b40eaeaa5] 2021-03-09T09:38:43.350 INFO:tasks.ceph.mon.b.smithi194.stderr: 11: (Context::complete(int)+0x9) [0x564b40eac059] 2021-03-09T09:38:43.350 INFO:tasks.ceph.mon.b.smithi194.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x564b40ed7548] 2021-03-09T09:38:43.350 INFO:tasks.ceph.mon.b.smithi194.stderr: 13: (Paxos::finish_round()+0x9b) [0x564b40fb27db] 2021-03-09T09:38:43.351 INFO:tasks.ceph.mon.b.smithi194.stderr: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x564b40fb3a06] 2021-03-09T09:38:43.351 INFO:tasks.ceph.mon.b.smithi194.stderr: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x564b40fb4633] 2021-03-09T09:38:43.351 INFO:tasks.ceph.mon.b.smithi194.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x564b40ea9de1] 2021-03-09T09:38:43.351 INFO:tasks.ceph.mon.b.smithi194.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x564b40eaa66a] 2021-03-09T09:38:43.352 INFO:tasks.ceph.mon.b.smithi194.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x564b40ed8178] 2021-03-09T09:38:43.352 INFO:tasks.ceph.mon.b.smithi194.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f5e1dbb42ca] 2021-03-09T09:38:43.352 INFO:tasks.ceph.mon.b.smithi194.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f5e1dc5e71d] 2021-03-09T09:38:43.352 INFO:tasks.ceph.mon.b.smithi194.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f5e1c8226db] 2021-03-09T09:38:43.352 INFO:tasks.ceph.mon.b.smithi194.stderr: 22: clone() 2021-03-09T09:38:43.353 INFO:tasks.ceph.mon.b.smithi194.stderr:*** Caught signal (Aborted) ** 2021-03-09T09:38:43.353 INFO:tasks.ceph.mon.b.smithi194.stderr: in thread 7f5e111f0700 thread_name:ms_dispatch 2021-03-09T09:38:43.353 INFO:tasks.ceph.mon.b.smithi194.stderr:2021-03-09T09:38:43.347+0000 7f5e111f0700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f5e111f0700 time 2021-03-09T09:38:43.333866+0000 2021-03-09T09:38:43.354 INFO:tasks.ceph.mon.b.smithi194.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:38:43.354 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.354 INFO:tasks.ceph.mon.b.smithi194.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:43.354 INFO:tasks.ceph.mon.b.smithi194.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5e1d9b23dd] 2021-03-09T09:38:43.355 INFO:tasks.ceph.mon.b.smithi194.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5e1d9b25b8] 2021-03-09T09:38:43.355 INFO:tasks.ceph.mon.b.smithi194.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x564b40fc7a65] 2021-03-09T09:38:43.355 INFO:tasks.ceph.mon.b.smithi194.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x564b41013a71] 2021-03-09T09:38:43.356 INFO:tasks.ceph.mon.b.smithi194.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x564b41043295] 2021-03-09T09:38:43.356 INFO:tasks.ceph.mon.b.smithi194.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x564b4104b24b] 2021-03-09T09:38:43.356 INFO:tasks.ceph.mon.b.smithi194.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x564b4104b7ab] 2021-03-09T09:38:43.357 INFO:tasks.ceph.mon.b.smithi194.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x564b40fbaade] 2021-03-09T09:38:43.357 INFO:tasks.ceph.mon.b.smithi194.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x564b40f10134] 2021-03-09T09:38:43.357 INFO:tasks.ceph.mon.b.smithi194.stderr: 10: (C_MonOp::finish(int)+0x45) [0x564b40eaeaa5] 2021-03-09T09:38:43.357 INFO:tasks.ceph.mon.b.smithi194.stderr: 11: (Context::complete(int)+0x9) [0x564b40eac059] 2021-03-09T09:38:43.357 INFO:tasks.ceph.mon.b.smithi194.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x564b40ed7548] 2021-03-09T09:38:43.358 INFO:tasks.ceph.mon.b.smithi194.stderr: 13: (Paxos::finish_round()+0x9b) [0x564b40fb27db] 2021-03-09T09:38:43.358 INFO:tasks.ceph.mon.b.smithi194.stderr: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x564b40fb3a06] 2021-03-09T09:38:43.358 INFO:tasks.ceph.mon.b.smithi194.stderr: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x564b40fb4633] 2021-03-09T09:38:43.358 INFO:tasks.ceph.mon.b.smithi194.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x564b40ea9de1] 2021-03-09T09:38:43.358 INFO:tasks.ceph.mon.b.smithi194.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x564b40eaa66a] 2021-03-09T09:38:43.359 INFO:tasks.ceph.mon.b.smithi194.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x564b40ed8178] 2021-03-09T09:38:43.359 INFO:tasks.ceph.mon.b.smithi194.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f5e1dbb42ca] 2021-03-09T09:38:43.359 INFO:tasks.ceph.mon.b.smithi194.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f5e1dc5e71d] 2021-03-09T09:38:43.359 INFO:tasks.ceph.mon.b.smithi194.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f5e1c8226db] 2021-03-09T09:38:43.360 INFO:tasks.ceph.mon.b.smithi194.stderr: 22: clone() 2021-03-09T09:38:43.360 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.360 INFO:tasks.ceph.mon.b.smithi194.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:43.361 INFO:tasks.ceph.mon.b.smithi194.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f5e1c82d980] 2021-03-09T09:38:43.361 INFO:tasks.ceph.mon.b.smithi194.stderr: 2: gsignal() 2021-03-09T09:38:43.361 INFO:tasks.ceph.mon.b.smithi194.stderr: 3: abort() 2021-03-09T09:38:43.362 INFO:tasks.ceph.mon.b.smithi194.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f5e1d9b242e] 2021-03-09T09:38:43.362 INFO:tasks.ceph.mon.b.smithi194.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5e1d9b25b8] 2021-03-09T09:38:43.362 INFO:tasks.ceph.mon.b.smithi194.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x564b40fc7a65] 2021-03-09T09:38:43.362 INFO:tasks.ceph.mon.b.smithi194.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x564b41013a71] 2021-03-09T09:38:43.363 INFO:tasks.ceph.mon.b.smithi194.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x564b41043295] 2021-03-09T09:38:43.363 INFO:tasks.ceph.mon.b.smithi194.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x564b4104b24b] 2021-03-09T09:38:43.363 INFO:tasks.ceph.mon.b.smithi194.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x564b4104b7ab] 2021-03-09T09:38:43.363 INFO:tasks.ceph.mon.b.smithi194.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x564b40fbaade] 2021-03-09T09:38:43.364 INFO:tasks.ceph.mon.b.smithi194.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x564b40f10134] 2021-03-09T09:38:43.364 INFO:tasks.ceph.mon.b.smithi194.stderr: 13: (C_MonOp::finish(int)+0x45) [0x564b40eaeaa5] 2021-03-09T09:38:43.364 INFO:tasks.ceph.mon.b.smithi194.stderr: 14: (Context::complete(int)+0x9) [0x564b40eac059] 2021-03-09T09:38:43.365 INFO:tasks.ceph.mon.b.smithi194.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x564b40ed7548] 2021-03-09T09:38:43.365 INFO:tasks.ceph.mon.b.smithi194.stderr: 16: (Paxos::finish_round()+0x9b) [0x564b40fb27db] 2021-03-09T09:38:43.365 INFO:tasks.ceph.mon.b.smithi194.stderr: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x564b40fb3a06] 2021-03-09T09:38:43.365 INFO:tasks.ceph.mon.b.smithi194.stderr: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x564b40fb4633] 2021-03-09T09:38:43.365 INFO:tasks.ceph.mon.b.smithi194.stderr: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x564b40ea9de1] 2021-03-09T09:38:43.366 INFO:tasks.ceph.mon.b.smithi194.stderr: 20: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x564b40eaa66a] 2021-03-09T09:38:43.366 INFO:tasks.ceph.mon.b.smithi194.stderr: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x564b40ed8178] 2021-03-09T09:38:43.366 INFO:tasks.ceph.mon.b.smithi194.stderr: 22: (DispatchQueue::entry()+0x11ea) [0x7f5e1dbb42ca] 2021-03-09T09:38:43.366 INFO:tasks.ceph.mon.b.smithi194.stderr: 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f5e1dc5e71d] 2021-03-09T09:38:43.367 INFO:tasks.ceph.mon.b.smithi194.stderr: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f5e1c8226db] 2021-03-09T09:38:43.367 INFO:tasks.ceph.mon.b.smithi194.stderr: 25: clone() 2021-03-09T09:38:43.367 INFO:tasks.ceph.mon.b.smithi194.stderr:2021-03-09T09:38:43.351+0000 7f5e111f0700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:38:43.367 INFO:tasks.ceph.mon.b.smithi194.stderr: in thread 7f5e111f0700 thread_name:ms_dispatch 2021-03-09T09:38:43.367 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.368 INFO:tasks.ceph.mon.b.smithi194.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:43.368 INFO:tasks.ceph.mon.b.smithi194.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f5e1c82d980] 2021-03-09T09:38:43.368 INFO:tasks.ceph.mon.b.smithi194.stderr: 2: gsignal() 2021-03-09T09:38:43.368 INFO:tasks.ceph.mon.b.smithi194.stderr: 3: abort() 2021-03-09T09:38:43.368 INFO:tasks.ceph.mon.b.smithi194.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f5e1d9b242e] 2021-03-09T09:38:43.369 INFO:tasks.ceph.mon.b.smithi194.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5e1d9b25b8] 2021-03-09T09:38:43.369 INFO:tasks.ceph.mon.b.smithi194.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x564b40fc7a65] 2021-03-09T09:38:43.369 INFO:tasks.ceph.mon.b.smithi194.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x564b41013a71] 2021-03-09T09:38:43.369 INFO:tasks.ceph.mon.b.smithi194.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x564b41043295] 2021-03-09T09:38:43.370 INFO:tasks.ceph.mon.b.smithi194.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x564b4104b24b] 2021-03-09T09:38:43.370 INFO:tasks.ceph.mon.b.smithi194.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x564b4104b7ab] 2021-03-09T09:38:43.370 INFO:tasks.ceph.mon.b.smithi194.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x564b40fbaade] 2021-03-09T09:38:43.370 INFO:tasks.ceph.mon.b.smithi194.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x564b40f10134] 2021-03-09T09:38:43.371 INFO:tasks.ceph.mon.b.smithi194.stderr: 13: (C_MonOp::finish(int)+0x45) [0x564b40eaeaa5] 2021-03-09T09:38:43.371 INFO:tasks.ceph.mon.b.smithi194.stderr: 14: (Context::complete(int)+0x9) [0x564b40eac059] 2021-03-09T09:38:43.371 INFO:tasks.ceph.mon.b.smithi194.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x564b40ed7548] 2021-03-09T09:38:43.371 INFO:tasks.ceph.mon.b.smithi194.stderr: 16: (Paxos::finish_round()+0x9b) [0x564b40fb27db] 2021-03-09T09:38:43.372 INFO:tasks.ceph.mon.b.smithi194.stderr: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x564b40fb3a06] 2021-03-09T09:38:43.372 INFO:tasks.ceph.mon.b.smithi194.stderr: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x564b40fb4633] 2021-03-09T09:38:43.372 INFO:tasks.ceph.mon.b.smithi194.stderr: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x564b40ea9de1] 2021-03-09T09:38:43.372 INFO:tasks.ceph.mon.b.smithi194.stderr: 20: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x564b40eaa66a] 2021-03-09T09:38:43.373 INFO:tasks.ceph.mon.b.smithi194.stderr: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x564b40ed8178] 2021-03-09T09:38:43.373 INFO:tasks.ceph.mon.b.smithi194.stderr: 22: (DispatchQueue::entry()+0x11ea) [0x7f5e1dbb42ca] 2021-03-09T09:38:43.373 INFO:tasks.ceph.mon.b.smithi194.stderr: 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f5e1dc5e71d] 2021-03-09T09:38:43.373 INFO:tasks.ceph.mon.b.smithi194.stderr: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f5e1c8226db] 2021-03-09T09:38:43.374 INFO:tasks.ceph.mon.b.smithi194.stderr: 25: clone() 2021-03-09T09:38:43.374 INFO:tasks.ceph.mon.b.smithi194.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:38:43.374 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.375 INFO:tasks.ceph.mon.b.smithi194.stderr: -1> 2021-03-09T09:38:43.347+0000 7f5e111f0700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f5e111f0700 time 2021-03-09T09:38:43.333866+0000 2021-03-09T09:38:43.375 INFO:tasks.ceph.mon.b.smithi194.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:38:43.376 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.376 INFO:tasks.ceph.mon.b.smithi194.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:43.376 INFO:tasks.ceph.mon.b.smithi194.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5e1d9b23dd] 2021-03-09T09:38:43.376 INFO:tasks.ceph.mon.b.smithi194.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5e1d9b25b8] 2021-03-09T09:38:43.377 INFO:tasks.ceph.mon.b.smithi194.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x564b40fc7a65] 2021-03-09T09:38:43.377 INFO:tasks.ceph.mon.b.smithi194.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x564b41013a71] 2021-03-09T09:38:43.377 INFO:tasks.ceph.mon.b.smithi194.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x564b41043295] 2021-03-09T09:38:43.377 INFO:tasks.ceph.mon.b.smithi194.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x564b4104b24b] 2021-03-09T09:38:43.378 INFO:tasks.ceph.mon.b.smithi194.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x564b4104b7ab] 2021-03-09T09:38:43.378 INFO:tasks.ceph.mon.b.smithi194.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x564b40fbaade] 2021-03-09T09:38:43.378 INFO:tasks.ceph.mon.b.smithi194.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x564b40f10134] 2021-03-09T09:38:43.378 INFO:tasks.ceph.mon.b.smithi194.stderr: 10: (C_MonOp::finish(int)+0x45) [0x564b40eaeaa5] 2021-03-09T09:38:43.378 INFO:tasks.ceph.mon.b.smithi194.stderr: 11: (Context::complete(int)+0x9) [0x564b40eac059] 2021-03-09T09:38:43.379 INFO:tasks.ceph.mon.b.smithi194.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x564b40ed7548] 2021-03-09T09:38:43.379 INFO:tasks.ceph.mon.b.smithi194.stderr: 13: (Paxos::finish_round()+0x9b) [0x564b40fb27db] 2021-03-09T09:38:43.379 INFO:tasks.ceph.mon.b.smithi194.stderr: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x564b40fb3a06] 2021-03-09T09:38:43.379 INFO:tasks.ceph.mon.b.smithi194.stderr: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x564b40fb4633] 2021-03-09T09:38:43.380 INFO:tasks.ceph.mon.b.smithi194.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x564b40ea9de1] 2021-03-09T09:38:43.380 INFO:tasks.ceph.mon.b.smithi194.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x564b40eaa66a] 2021-03-09T09:38:43.380 INFO:tasks.ceph.mon.b.smithi194.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x564b40ed8178] 2021-03-09T09:38:43.380 INFO:tasks.ceph.mon.b.smithi194.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f5e1dbb42ca] 2021-03-09T09:38:43.380 INFO:tasks.ceph.mon.b.smithi194.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f5e1dc5e71d] 2021-03-09T09:38:43.381 INFO:tasks.ceph.mon.b.smithi194.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f5e1c8226db] 2021-03-09T09:38:43.381 INFO:tasks.ceph.mon.b.smithi194.stderr: 22: clone() 2021-03-09T09:38:43.381 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.382 INFO:tasks.ceph.mon.b.smithi194.stderr: 0> 2021-03-09T09:38:43.351+0000 7f5e111f0700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:38:43.382 INFO:tasks.ceph.mon.b.smithi194.stderr: in thread 7f5e111f0700 thread_name:ms_dispatch 2021-03-09T09:38:43.382 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.382 INFO:tasks.ceph.mon.b.smithi194.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:43.382 INFO:tasks.ceph.mon.b.smithi194.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f5e1c82d980] 2021-03-09T09:38:43.383 INFO:tasks.ceph.mon.b.smithi194.stderr: 2: gsignal() 2021-03-09T09:38:43.383 INFO:tasks.ceph.mon.b.smithi194.stderr: 3: abort() 2021-03-09T09:38:43.383 INFO:tasks.ceph.mon.b.smithi194.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f5e1d9b242e] 2021-03-09T09:38:43.383 INFO:tasks.ceph.mon.b.smithi194.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5e1d9b25b8] 2021-03-09T09:38:43.384 INFO:tasks.ceph.mon.b.smithi194.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x564b40fc7a65] 2021-03-09T09:38:43.384 INFO:tasks.ceph.mon.b.smithi194.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x564b41013a71] 2021-03-09T09:38:43.384 INFO:tasks.ceph.mon.b.smithi194.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x564b41043295] 2021-03-09T09:38:43.384 INFO:tasks.ceph.mon.b.smithi194.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x564b4104b24b] 2021-03-09T09:38:43.385 INFO:tasks.ceph.mon.b.smithi194.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x564b4104b7ab] 2021-03-09T09:38:43.385 INFO:tasks.ceph.mon.b.smithi194.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x564b40fbaade] 2021-03-09T09:38:43.385 INFO:tasks.ceph.mon.b.smithi194.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x564b40f10134] 2021-03-09T09:38:43.385 INFO:tasks.ceph.mon.b.smithi194.stderr: 13: (C_MonOp::finish(int)+0x45) [0x564b40eaeaa5] 2021-03-09T09:38:43.386 INFO:tasks.ceph.mon.b.smithi194.stderr: 14: (Context::complete(int)+0x9) [0x564b40eac059] 2021-03-09T09:38:43.386 INFO:tasks.ceph.mon.b.smithi194.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x564b40ed7548] 2021-03-09T09:38:43.386 INFO:tasks.ceph.mon.b.smithi194.stderr: 16: (Paxos::finish_round()+0x9b) [0x564b40fb27db] 2021-03-09T09:38:43.387 INFO:tasks.ceph.mon.b.smithi194.stderr: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x564b40fb3a06] 2021-03-09T09:38:43.387 INFO:tasks.ceph.mon.b.smithi194.stderr: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x564b40fb4633] 2021-03-09T09:38:43.387 INFO:tasks.ceph.mon.b.smithi194.stderr: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x564b40ea9de1] 2021-03-09T09:38:43.387 INFO:tasks.ceph.mon.b.smithi194.stderr: 20: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x564b40eaa66a] 2021-03-09T09:38:43.387 INFO:tasks.ceph.mon.b.smithi194.stderr: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x564b40ed8178] 2021-03-09T09:38:43.388 INFO:tasks.ceph.mon.b.smithi194.stderr: 22: (DispatchQueue::entry()+0x11ea) [0x7f5e1dbb42ca] 2021-03-09T09:38:43.388 INFO:tasks.ceph.mon.b.smithi194.stderr: 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f5e1dc5e71d] 2021-03-09T09:38:43.389 INFO:tasks.ceph.mon.b.smithi194.stderr: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f5e1c8226db] 2021-03-09T09:38:43.389 INFO:tasks.ceph.mon.b.smithi194.stderr: 25: clone() 2021-03-09T09:38:43.389 INFO:tasks.ceph.mon.b.smithi194.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:38:43.390 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.391 INFO:tasks.ceph.mon.b.smithi194.stderr: -1> 2021-03-09T09:38:43.347+0000 7f5e111f0700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f5e111f0700 time 2021-03-09T09:38:43.333866+0000 2021-03-09T09:38:43.391 INFO:tasks.ceph.mon.b.smithi194.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:38:43.392 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.392 INFO:tasks.ceph.mon.b.smithi194.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:43.393 INFO:tasks.ceph.mon.b.smithi194.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5e1d9b23dd] 2021-03-09T09:38:43.393 INFO:tasks.ceph.mon.b.smithi194.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5e1d9b25b8] 2021-03-09T09:38:43.394 INFO:tasks.ceph.mon.b.smithi194.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x564b40fc7a65] 2021-03-09T09:38:43.394 INFO:tasks.ceph.mon.b.smithi194.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x564b41013a71] 2021-03-09T09:38:43.394 INFO:tasks.ceph.mon.b.smithi194.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x564b41043295] 2021-03-09T09:38:43.395 INFO:tasks.ceph.mon.b.smithi194.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x564b4104b24b] 2021-03-09T09:38:43.395 INFO:tasks.ceph.mon.b.smithi194.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x564b4104b7ab] 2021-03-09T09:38:43.396 INFO:tasks.ceph.mon.b.smithi194.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x564b40fbaade] 2021-03-09T09:38:43.396 INFO:tasks.ceph.mon.b.smithi194.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x564b40f10134] 2021-03-09T09:38:43.397 INFO:tasks.ceph.mon.b.smithi194.stderr: 10: (C_MonOp::finish(int)+0x45) [0x564b40eaeaa5] 2021-03-09T09:38:43.397 INFO:tasks.ceph.mon.b.smithi194.stderr: 11: (Context::complete(int)+0x9) [0x564b40eac059] 2021-03-09T09:38:43.397 INFO:tasks.ceph.mon.b.smithi194.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x564b40ed7548] 2021-03-09T09:38:43.398 INFO:tasks.ceph.mon.b.smithi194.stderr: 13: (Paxos::finish_round()+0x9b) [0x564b40fb27db] 2021-03-09T09:38:43.398 INFO:tasks.ceph.mon.b.smithi194.stderr: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x564b40fb3a06] 2021-03-09T09:38:43.399 INFO:tasks.ceph.mon.b.smithi194.stderr: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x564b40fb4633] 2021-03-09T09:38:43.399 INFO:tasks.ceph.mon.b.smithi194.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x564b40ea9de1] 2021-03-09T09:38:43.399 INFO:tasks.ceph.mon.b.smithi194.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x564b40eaa66a] 2021-03-09T09:38:43.400 INFO:tasks.ceph.mon.b.smithi194.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x564b40ed8178] 2021-03-09T09:38:43.400 INFO:tasks.ceph.mon.b.smithi194.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f5e1dbb42ca] 2021-03-09T09:38:43.401 INFO:tasks.ceph.mon.b.smithi194.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f5e1dc5e71d] 2021-03-09T09:38:43.401 INFO:tasks.ceph.mon.b.smithi194.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f5e1c8226db] 2021-03-09T09:38:43.401 INFO:tasks.ceph.mon.b.smithi194.stderr: 22: clone() 2021-03-09T09:38:43.402 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.402 INFO:tasks.ceph.mon.b.smithi194.stderr: 0> 2021-03-09T09:38:43.351+0000 7f5e111f0700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:38:43.403 INFO:tasks.ceph.mon.b.smithi194.stderr: in thread 7f5e111f0700 thread_name:ms_dispatch 2021-03-09T09:38:43.403 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:43.404 INFO:tasks.ceph.mon.b.smithi194.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:38:43.404 INFO:tasks.ceph.mon.b.smithi194.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f5e1c82d980] 2021-03-09T09:38:43.404 INFO:tasks.ceph.mon.b.smithi194.stderr: 2: gsignal() 2021-03-09T09:38:43.405 INFO:tasks.ceph.mon.b.smithi194.stderr: 3: abort() 2021-03-09T09:38:43.405 INFO:tasks.ceph.mon.b.smithi194.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f5e1d9b242e] 2021-03-09T09:38:43.406 INFO:tasks.ceph.mon.b.smithi194.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5e1d9b25b8] 2021-03-09T09:38:43.406 INFO:tasks.ceph.mon.b.smithi194.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x564b40fc7a65] 2021-03-09T09:38:43.406 INFO:tasks.ceph.mon.b.smithi194.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x564b41013a71] 2021-03-09T09:38:43.407 INFO:tasks.ceph.mon.b.smithi194.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x564b41043295] 2021-03-09T09:38:43.407 INFO:tasks.ceph.mon.b.smithi194.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x564b4104b24b] 2021-03-09T09:38:43.408 INFO:tasks.ceph.mon.b.smithi194.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x564b4104b7ab] 2021-03-09T09:38:43.408 INFO:tasks.ceph.mon.b.smithi194.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x564b40fbaade] 2021-03-09T09:38:43.409 INFO:tasks.ceph.mon.b.smithi194.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x564b40f10134] 2021-03-09T09:38:43.409 INFO:tasks.ceph.mon.b.smithi194.stderr: 13: (C_MonOp::finish(int)+0x45) [0x564b40eaeaa5] 2021-03-09T09:38:43.409 INFO:tasks.ceph.mon.b.smithi194.stderr: 14: (Context::complete(int)+0x9) [0x564b40eac059] 2021-03-09T09:38:43.410 INFO:tasks.ceph.mon.b.smithi194.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x564b40ed7548] 2021-03-09T09:38:43.410 INFO:tasks.ceph.mon.b.smithi194.stderr: 16: (Paxos::finish_round()+0x9b) [0x564b40fb27db] 2021-03-09T09:38:43.411 INFO:tasks.ceph.mon.b.smithi194.stderr: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x564b40fb3a06] 2021-03-09T09:38:43.411 INFO:tasks.ceph.mon.b.smithi194.stderr: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x564b40fb4633] 2021-03-09T09:38:43.412 INFO:tasks.ceph.mon.b.smithi194.stderr: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x564b40ea9de1] 2021-03-09T09:38:43.412 INFO:tasks.ceph.mon.b.smithi194.stderr: 20: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x564b40eaa66a] 2021-03-09T09:38:43.412 INFO:tasks.ceph.mon.b.smithi194.stderr: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x564b40ed8178] 2021-03-09T09:38:43.413 INFO:tasks.ceph.mon.b.smithi194.stderr: 22: (DispatchQueue::entry()+0x11ea) [0x7f5e1dbb42ca] 2021-03-09T09:38:43.413 INFO:tasks.ceph.mon.b.smithi194.stderr: 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f5e1dc5e71d] 2021-03-09T09:38:43.414 INFO:tasks.ceph.mon.b.smithi194.stderr: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f5e1c8226db] 2021-03-09T09:38:43.414 INFO:tasks.ceph.mon.b.smithi194.stderr: 25: clone() 2021-03-09T09:38:43.415 INFO:tasks.ceph.mon.b.smithi194.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:38:43.415 INFO:tasks.ceph.mon.b.smithi194.stderr: 2021-03-09T09:38:44.352 INFO:tasks.ceph.mon.b.smithi194.stderr:daemon-helper: command crashed with signal 6 2021-03-09T09:38:49.686 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T09:38:49.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 Processor -- start 2021-03-09T09:38:49.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 -- start start 2021-03-09T09:38:49.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c0adc20 0x7ff99c0aaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:49.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99c140b10 con 0x7ff99c0613a0 2021-03-09T09:38:49.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99c0aaf70 con 0x7ff99c141790 2021-03-09T09:38:49.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c0ae000 con 0x7ff99c0adc20 2021-03-09T09:38:49.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99affd700 1 -- >> v1:172.21.15.52:6789/0 conn(0x7ff99c0613a0 legacy=0x7ff99c141080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2021-03-09T09:38:49.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99b7fe700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c0adc20 0x7ff99c0aaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:49.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99b7fe700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c0adc20 0x7ff99c0aaa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3301/0 says I am v2:172.21.15.52:35172/0 (socket says 172.21.15.52:35172) 2021-03-09T09:38:49.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99b7fe700 1 -- 172.21.15.52:0/279550165 learned_addr learned my addr 172.21.15.52:0/279550165 (peer_addr_for_me v2:172.21.15.52:0/0) 2021-03-09T09:38:49.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/279550165 >> v1:172.21.15.194:6789/0 conn(0x7ff99c141790 legacy=0x7ff99c147eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2021-03-09T09:38:49.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99b7fe700 1 -- 172.21.15.52:0/279550165 >> v1:172.21.15.194:6789/0 conn(0x7ff99c141790 legacy=0x7ff99c147eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T09:38:49.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99b7fe700 1 -- 172.21.15.52:0/279550165 >> v1:172.21.15.52:6789/0 conn(0x7ff99c0613a0 legacy=0x7ff99c141080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T09:38:49.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99b7fe700 1 -- 172.21.15.52:0/279550165 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c140d40 con 0x7ff99c0adc20 2021-03-09T09:38:49.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/279550165 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c0adc20 0x7ff99c0aaa80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff994009290).ready entity=mon.2 client_cookie=8c67ac0022fe1390 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:49.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99a7fc700 1 -- 172.21.15.52:0/279550165 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400bc00 con 0x7ff99c0adc20 2021-03-09T09:38:49.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99a7fc700 1 -- 172.21.15.52:0/279550165 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400bd60 con 0x7ff99c0adc20 2021-03-09T09:38:49.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff99a7fc700 1 -- 172.21.15.52:0/279550165 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994011610 con 0x7ff99c0adc20 2021-03-09T09:38:49.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 -- 172.21.15.52:0/279550165 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c0adc20 msgr2=0x7ff99c0aaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:38:49.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 --2- 172.21.15.52:0/279550165 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c0adc20 0x7ff99c0aaa80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff994009290).stop 2021-03-09T09:38:49.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 -- 172.21.15.52:0/279550165 shutdown_connections 2021-03-09T09:38:49.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 --2- 172.21.15.52:0/279550165 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c0adc20 0x7ff99c0aaa80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:49.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 -- 172.21.15.52:0/279550165 >> 172.21.15.52:0/279550165 conn(0x7ff99c065f70 msgr2=0x7ff99c066380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:38:49.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 -- 172.21.15.52:0/279550165 shutdown_connections 2021-03-09T09:38:49.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.879+0000 7ff9a20e7700 1 -- 172.21.15.52:0/279550165 wait complete. 2021-03-09T09:38:49.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff9a20e7700 1 Processor -- start 2021-03-09T09:38:49.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff9a20e7700 1 -- start start 2021-03-09T09:38:49.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff9a20e7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff99c0a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:49.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff9a20e7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff99c090b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:49.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff9a20e7700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff99c150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:49.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff9a20e7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c091f70 con 0x7ff99c0adc20 2021-03-09T09:38:49.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff9a20e7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c0921a0 con 0x7ff99c0613a0 2021-03-09T09:38:49.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff9a20e7700 1 -- --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c092440 con 0x7ff99c141790 2021-03-09T09:38:49.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99affd700 1 -- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff99c090b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:38:49.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff99c090b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:49.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99b7fe700 1 -- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff99c0a0210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:38:49.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff99c0a0210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:49.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99bfff700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff99c150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:49.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99bfff700 1 --2- >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff99c150640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3301/0 says I am v2:172.21.15.52:35182/0 (socket says 172.21.15.52:35182) 2021-03-09T09:38:49.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 learned_addr learned my addr 172.21.15.52:0/1490464907 (peer_addr_for_me v2:172.21.15.52:0/0) 2021-03-09T09:38:49.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff99c0a0210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:49.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff99c0a0210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:49.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff99c090b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:49.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff99c090b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:49.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c0ae6a0 con 0x7ff99c141790 2021-03-09T09:38:49.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.883+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff99c150640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7ff98c009690 tx=0x7ff98c006d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c006170 con 0x7ff99c141790 2021-03-09T09:38:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0062d0 con 0x7ff99c141790 2021-03-09T09:38:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.887+0000 7ff9a20e7700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99c13df80 con 0x7ff99c141790 2021-03-09T09:38:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c003880 con 0x7ff99c141790 2021-03-09T09:38:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.887+0000 7ff9a20e7700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99c0a0eb0 con 0x7ff99c141790 2021-03-09T09:38:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.887+0000 7ff9a20e7700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99c05e210 con 0x7ff99c141790 2021-03-09T09:38:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 4 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00f070 con 0x7ff99c141790 2021-03-09T09:38:49.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.891+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:49.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.891+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:49.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.891+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:38:49.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.891+0000 7ff9937fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:38:49.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.891+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800855c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:49.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800855c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:49.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 2212+0+0 (secure 0 0 0) 0x7ff98c0a40f0 con 0x7ff99c141790 2021-03-09T09:38:49.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800855c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7ff9940092d0 tx=0x7ff994017000).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:49.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:49.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+149001 (secure 0 0 0) 0x7ff98c0791c0 con 0x7ff99c141790 2021-03-09T09:38:50.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:38:50.075+0000 7ff9a20e7700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff99c09d5a0 con 0x7ff99c141790 2021-03-09T09:38:54.144 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:38:54.147 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:38:58.962 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:38:58.959+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T09:38:27.132037+0000) 2021-03-09T09:39:03.962 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:03.963+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:03.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 msgr2=0x7ff99c150640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:03.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 msgr2=0x7ff99c150640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:03.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff99c150640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7ff98c009690 tx=0x7ff98c006d50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:03.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff99c150640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7ff98c009690 tx=0x7ff98c006d50).stop 2021-03-09T09:39:03.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 msgr2=0x7ff99c150640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:03.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff99c150640 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:03.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980089e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:03.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:03.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:03.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008c460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:03.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:03.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ef10 con 0x7ff99c0adc20 2021-03-09T09:39:03.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008f050 con 0x7ff99c0613a0 2021-03-09T09:39:03.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008ea00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:03.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008f190 con 0x7ff980087a10 2021-03-09T09:39:03.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:03.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980089e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:03.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008c460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:03.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.963+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:03.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.967+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008ea00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:03.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.967+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:03.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.967+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008f320 con 0x7ff980087a10 2021-03-09T09:39:03.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.967+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9840096b0 con 0x7ff980087a10 2021-03-09T09:39:03.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.967+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980089e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ff9840099a0 tx=0x7ff984009c90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:03.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.967+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98400cb10 con 0x7ff980087a10 2021-03-09T09:39:03.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.971+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400cc70 con 0x7ff980087a10 2021-03-09T09:39:03.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.975+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984003670 con 0x7ff980087a10 2021-03-09T09:39:03.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.975+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:03.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.979+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:03.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:03.979+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:08.963 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:08.959+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:08.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.959+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980089e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:08.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.959+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980089e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:08.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.959+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980089e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ff9840099a0 tx=0x7ff984009c90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:08.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.959+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980089e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ff9840099a0 tx=0x7ff984009c90).stop 2021-03-09T09:39:08.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.959+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980089e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:08.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.959+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980089e20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:08.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.959+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:08.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.959+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff980029dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:08.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:08.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a2b0 con 0x7ff99c0adc20 2021-03-09T09:39:08.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a3f0 con 0x7ff99c0613a0 2021-03-09T09:39:08.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008c460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:08.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a530 con 0x7ff99c141790 2021-03-09T09:39:08.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:08.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008ea00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:08.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:08.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff980029dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:08.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008c460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:08.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:08.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008ea00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:08.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:08.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091610 con 0x7ff99c141790 2021-03-09T09:39:08.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff99c09d5a0 con 0x7ff99c141790 2021-03-09T09:39:08.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.963+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff980029dc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff99c0277c0 tx=0x7ff98c0a8000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:08.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.967+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c090070 con 0x7ff99c141790 2021-03-09T09:39:08.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.971+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079590 con 0x7ff99c141790 2021-03-09T09:39:08.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.975+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c003bf0 con 0x7ff99c141790 2021-03-09T09:39:08.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.975+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:08.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.975+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:08.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:08.975+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:13.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 msgr2=0x7ff980029dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:13.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 msgr2=0x7ff980029dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:13.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff980029dc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff99c0277c0 tx=0x7ff98c0a8000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:13.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff980029dc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff99c0277c0 tx=0x7ff98c0a8000).stop 2021-03-09T09:39:13.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 msgr2=0x7ff980029dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:13.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff980029dc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:13.967 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:13.961+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:13.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:13.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:13.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008ea00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:13.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:13.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:13.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002ea70 con 0x7ff99c0adc20 2021-03-09T09:39:13.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980031ea0 con 0x7ff99c0613a0 2021-03-09T09:39:13.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003210 con 0x7ff980087a10 2021-03-09T09:39:13.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008c460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:13.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:13.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:13.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008c460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:13.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:13.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008ea00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:13.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:13.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980053ac0 con 0x7ff980087a10 2021-03-09T09:39:13.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff99c140d40 con 0x7ff980087a10 2021-03-09T09:39:13.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.965+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800535d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff994011000 tx=0x7ff99400f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:13.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.965+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994030470 con 0x7ff980087a10 2021-03-09T09:39:13.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.965+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940305d0 con 0x7ff980087a10 2021-03-09T09:39:13.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99400f070 con 0x7ff980087a10 2021-03-09T09:39:13.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.969+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:13.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.969+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:13.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:13.969+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:18.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800535d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:18.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800535d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:18.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800535d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff994011000 tx=0x7ff99400f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:18.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800535d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff994011000 tx=0x7ff99400f000).stop 2021-03-09T09:39:18.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800535d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:18.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800535d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:18.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:18.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:18.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff9800917a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:18.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e920 con 0x7ff99c0adc20 2021-03-09T09:39:18.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008d80 con 0x7ff99c0613a0 2021-03-09T09:39:18.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800365e0 con 0x7ff99c141790 2021-03-09T09:39:18.971 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:18.964+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:18.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c9f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:18.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:18.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:18.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:18.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff9800917a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:18.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:18.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:18.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:18.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:18.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091c90 con 0x7ff99c141790 2021-03-09T09:39:18.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff99c140d40 con 0x7ff99c141790 2021-03-09T09:39:18.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff9800917a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff994081b90 tx=0x7ff994081670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:18.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99401c960 con 0x7ff99c141790 2021-03-09T09:39:18.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086000 con 0x7ff99c141790 2021-03-09T09:39:18.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994030950 con 0x7ff99c141790 2021-03-09T09:39:18.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.972+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:18.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.972+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:18.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:18.972+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:23.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 msgr2=0x7ff9800917a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:39:23.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 msgr2=0x7ff9800917a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:23.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff9800917a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff994081b90 tx=0x7ff994081670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:23.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff9800917a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff994081b90 tx=0x7ff994081670).stop 2021-03-09T09:39:23.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 msgr2=0x7ff9800917a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:23.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff99c141790 0x7ff9800917a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:23.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:23.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:23.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:23.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050f10 con 0x7ff99c0adc20 2021-03-09T09:39:23.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980031d30 con 0x7ff99c0613a0 2021-03-09T09:39:23.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980007160 con 0x7ff980087a10 2021-03-09T09:39:23.972 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:23.964+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:23.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:23.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:23.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:23.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:23.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:23.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:23.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:23.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:23.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:23.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004d350 con 0x7ff980087a10 2021-03-09T09:39:23.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff99c140d40 con 0x7ff980087a10 2021-03-09T09:39:23.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004ce60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ff980068390 tx=0x7ff99405cc40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:23.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405c830 con 0x7ff980087a10 2021-03-09T09:39:23.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405e070 con 0x7ff980087a10 2021-03-09T09:39:23.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994030be0 con 0x7ff980087a10 2021-03-09T09:39:23.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.976+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:23.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.976+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:23.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:23.976+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:24.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:39:24.311 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:39:28.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98004ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:39:28.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98004ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:28.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004ce60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ff980068390 tx=0x7ff99405cc40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:28.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004ce60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ff980068390 tx=0x7ff99405cc40).stop 2021-03-09T09:39:28.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98004ce60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:28.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004ce60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:28.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:28.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:28.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800200c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:28.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800914b0 con 0x7ff99c0adc20 2021-03-09T09:39:28.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008a1b0 con 0x7ff99c0613a0 2021-03-09T09:39:28.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800205b0 con 0x7ff98001dc30 2021-03-09T09:39:28.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:28.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:28.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:28.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.964+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:28.974 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:28.964+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:28.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800200c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:28.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:28.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:28.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:28.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:28.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800206f0 con 0x7ff98001dc30 2021-03-09T09:39:28.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff99c140d40 con 0x7ff98001dc30 2021-03-09T09:39:28.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800200c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff994036350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:28.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99403a8a0 con 0x7ff98001dc30 2021-03-09T09:39:28.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081020 con 0x7ff98001dc30 2021-03-09T09:39:28.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940320d0 con 0x7ff98001dc30 2021-03-09T09:39:28.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.976+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:28.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.976+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:28.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:28.976+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:33.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800200c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:33.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800200c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:33.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800200c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff994036350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:33.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800200c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff994036350).stop 2021-03-09T09:39:33.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800200c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:33.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800200c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:33.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:33.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:33.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980047ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:33.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800205b0 con 0x7ff99c0adc20 2021-03-09T09:39:33.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:33.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053970 con 0x7ff99c0613a0 2021-03-09T09:39:33.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091b30 con 0x7ff980087a10 2021-03-09T09:39:33.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:33.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:33.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.964+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:33.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980047ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:33.975 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:33.968+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 68 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:33.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:33.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:33.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:33.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:33.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800819f0 con 0x7ff980087a10 2021-03-09T09:39:33.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800206f0 con 0x7ff980087a10 2021-03-09T09:39:33.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980047ca0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff980068390 tx=0x7ff994032440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:33.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a200 con 0x7ff980087a10 2021-03-09T09:39:33.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407de90 con 0x7ff980087a10 2021-03-09T09:39:33.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99407ee70 con 0x7ff980087a10 2021-03-09T09:39:33.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.976+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:33.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.976+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:33.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:33.976+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:38.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980047ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:38.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980047ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:38.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980047ca0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff980068390 tx=0x7ff994032440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:38.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980047ca0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff980068390 tx=0x7ff994032440).stop 2021-03-09T09:39:38.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980047ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:38.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980047ca0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:38.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:38.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:38.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:38.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980031be0 con 0x7ff99c0adc20 2021-03-09T09:39:38.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800306b0 con 0x7ff99c0613a0 2021-03-09T09:39:38.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980087800 con 0x7ff98001dc30 2021-03-09T09:39:38.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:38.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:38.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:38.975 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:38.968+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 102 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:38.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:38.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:38.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:38.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:38.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:38.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:38.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050ec0 con 0x7ff98001dc30 2021-03-09T09:39:38.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.968+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9840096b0 con 0x7ff98001dc30 2021-03-09T09:39:38.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001b250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff98407b9e0 tx=0x7ff984009620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:38.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98407b590 con 0x7ff98001dc30 2021-03-09T09:39:38.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400b130 con 0x7ff98001dc30 2021-03-09T09:39:38.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff98001dc30 2021-03-09T09:39:38.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.976+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:38.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.976+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:38.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:38.976+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:43.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98001b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:43.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98001b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:43.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001b250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff98407b9e0 tx=0x7ff984009620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:43.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001b250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff98407b9e0 tx=0x7ff984009620).stop 2021-03-09T09:39:43.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98001b250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:43.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001b250 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:43.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:43.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:43.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:43.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98007c250 con 0x7ff99c0adc20 2021-03-09T09:39:43.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001d560 con 0x7ff99c0613a0 2021-03-09T09:39:43.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018b60 con 0x7ff980087a10 2021-03-09T09:39:43.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:43.975 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:43.968+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 133 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:43.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:43.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:43.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:43.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:43.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:43.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:43.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:43.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:43.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091480 con 0x7ff980087a10 2021-03-09T09:39:43.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800819f0 con 0x7ff980087a10 2021-03-09T09:39:43.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800442c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ff994006fd0 tx=0x7ff994072be0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:43.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994033c50 con 0x7ff980087a10 2021-03-09T09:39:43.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400b750 con 0x7ff980087a10 2021-03-09T09:39:43.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980087a10 2021-03-09T09:39:43.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.976+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:43.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.980+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:43.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:43.980+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:48.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800442c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:39:48.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800442c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:48.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800442c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ff994006fd0 tx=0x7ff994072be0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:48.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800442c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ff994006fd0 tx=0x7ff994072be0).stop 2021-03-09T09:39:48.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800442c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:48.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800442c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:48.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:48.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:48.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:48.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98007e910 con 0x7ff99c0adc20 2021-03-09T09:39:48.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800821d0 con 0x7ff99c0613a0 2021-03-09T09:39:48.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98007c250 con 0x7ff98001dc30 2021-03-09T09:39:48.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:48.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:48.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:48.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.968+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:48.978 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:48.968+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 164 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:48.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:48.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.972+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:48.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:48.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.972+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:48.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:48.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.972+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980023df0 con 0x7ff98001dc30 2021-03-09T09:39:48.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.972+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9840096b0 con 0x7ff98001dc30 2021-03-09T09:39:48.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984009d00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:48.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff98001dc30 2021-03-09T09:39:48.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400c900 con 0x7ff98001dc30 2021-03-09T09:39:48.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98402bc40 con 0x7ff98001dc30 2021-03-09T09:39:48.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.980+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:48.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.980+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:48.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:48.980+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:53.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.968+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980093e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:39:53.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.968+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980093e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:53.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.968+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984009d00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:53.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984009d00).stop 2021-03-09T09:39:53.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980093e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:53.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:53.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980040c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:53.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:53.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:53.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043be0 con 0x7ff99c0adc20 2021-03-09T09:39:53.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98007b080 con 0x7ff99c0613a0 2021-03-09T09:39:53.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004c710 con 0x7ff980087a10 2021-03-09T09:39:53.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:53.978 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:53.972+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 196 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:53.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:53.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:53.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:53.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980040c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:53.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:53.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:53.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:53.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:53.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980041180 con 0x7ff980087a10 2021-03-09T09:39:53.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9840096b0 con 0x7ff980087a10 2021-03-09T09:39:53.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980040c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7ff984058a60 tx=0x7ff9840582f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:53.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980087a10 2021-03-09T09:39:53.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984058670 con 0x7ff980087a10 2021-03-09T09:39:53.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984030400 con 0x7ff980087a10 2021-03-09T09:39:53.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.984+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:53.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:53.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:53.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:39:54.471 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:39:54.474 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:39:58.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980040c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:58.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980040c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:58.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980040c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7ff984058a60 tx=0x7ff9840582f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:58.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980040c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7ff984058a60 tx=0x7ff9840582f0).stop 2021-03-09T09:39:58.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980040c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:58.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980040c90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:58.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:58.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:58.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:58.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98007b7f0 con 0x7ff99c0adc20 2021-03-09T09:39:58.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008c30 con 0x7ff99c0613a0 2021-03-09T09:39:58.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003e5c0 con 0x7ff98001dc30 2021-03-09T09:39:58.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:39:58.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:58.979 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:39:58.972+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 232 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:39:58.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:39:58.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:58.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:58.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:58.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:58.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:58.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:58.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094320 con 0x7ff98001dc30 2021-03-09T09:39:58.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091480 con 0x7ff98001dc30 2021-03-09T09:39:58.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff980074990 tx=0x7ff99406a180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:58.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994070260 con 0x7ff98001dc30 2021-03-09T09:39:58.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940703c0 con 0x7ff98001dc30 2021-03-09T09:39:58.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99405c990 con 0x7ff98001dc30 2021-03-09T09:39:58.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.984+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:58.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:58.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:39:58.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:03.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980093e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:03.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980093e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:03.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff980074990 tx=0x7ff99406a180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:03.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff980074990 tx=0x7ff99406a180).stop 2021-03-09T09:40:03.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980093e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:03.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980093e30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:03.977 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:03.972+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 269 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:03.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:03.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:03.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:03.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98007b7f0 con 0x7ff99c0adc20 2021-03-09T09:40:03.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98007de60 con 0x7ff99c0613a0 2021-03-09T09:40:03.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980076300 con 0x7ff980087a10 2021-03-09T09:40:03.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:03.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:03.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:03.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:03.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.972+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:03.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:03.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:03.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:03.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:03.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a950 con 0x7ff980087a10 2021-03-09T09:40:03.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff99c09d5a0 con 0x7ff980087a10 2021-03-09T09:40:03.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003a460 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff98c028e70 tx=0x7ff98c028af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:03.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023070 con 0x7ff980087a10 2021-03-09T09:40:03.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0791c0 con 0x7ff980087a10 2021-03-09T09:40:03.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980087a10 2021-03-09T09:40:03.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.980+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:03.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:03.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:03.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:08.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98003a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:08.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98003a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:08.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003a460 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff98c028e70 tx=0x7ff98c028af0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:08.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003a460 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff98c028e70 tx=0x7ff98c028af0).stop 2021-03-09T09:40:08.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98003a460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:08.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003a460 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:08.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:08.978 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:08.972+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 303 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:08.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:08.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:08.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980076300 con 0x7ff99c0adc20 2021-03-09T09:40:08.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980007890 con 0x7ff99c0613a0 2021-03-09T09:40:08.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033820 con 0x7ff98001dc30 2021-03-09T09:40:08.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:08.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:08.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:08.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:08.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:08.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:08.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:08.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:08.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:08.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000e4f0 con 0x7ff98001dc30 2021-03-09T09:40:08.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094320 con 0x7ff98001dc30 2021-03-09T09:40:08.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e000 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406bf70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:08.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994065bf0 con 0x7ff98001dc30 2021-03-09T09:40:08.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994065bf0 con 0x7ff98001dc30 2021-03-09T09:40:08.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99406b050 con 0x7ff98001dc30 2021-03-09T09:40:08.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.984+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:08.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.988+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:08.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:08.988+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:13.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98000e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:40:13.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98000e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:13.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e000 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406bf70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:13.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e000 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406bf70).stop 2021-03-09T09:40:13.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98000e000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:13.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e000 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:13.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:13.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800944b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:13.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:13.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980076300 con 0x7ff99c0adc20 2021-03-09T09:40:13.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029710 con 0x7ff99c0613a0 2021-03-09T09:40:13.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023ca0 con 0x7ff980087a10 2021-03-09T09:40:13.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:13.980 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:13.972+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 341 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:13.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:13.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:13.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:13.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.972+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800944b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:13.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:13.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:13.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:13.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:13.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004e620 con 0x7ff980087a10 2021-03-09T09:40:13.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003a950 con 0x7ff980087a10 2021-03-09T09:40:13.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800944b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff9800777a0 tx=0x7ff98c07a3a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:13.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023070 con 0x7ff980087a10 2021-03-09T09:40:13.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980087a10 2021-03-09T09:40:13.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a33b0 con 0x7ff980087a10 2021-03-09T09:40:13.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.984+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:13.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:13.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:13.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:18.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800944b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:18.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800944b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:18.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800944b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff9800777a0 tx=0x7ff98c07a3a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:18.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800944b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff9800777a0 tx=0x7ff98c07a3a0).stop 2021-03-09T09:40:18.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800944b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:18.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800944b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:18.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:18.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:18.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:18.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980007890 con 0x7ff99c0adc20 2021-03-09T09:40:18.982 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:18.972+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 375 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:18.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000fff0 con 0x7ff99c0613a0 2021-03-09T09:40:18.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.972+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004eb0 con 0x7ff98001dc30 2021-03-09T09:40:18.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:18.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:18.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:18.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:18.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:18.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:18.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:18.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:18.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:18.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980039ca0 con 0x7ff98001dc30 2021-03-09T09:40:18.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9840096b0 con 0x7ff98001dc30 2021-03-09T09:40:18.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004aed0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff9840813b0 tx=0x7ff98402c120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:18.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019040 con 0x7ff98001dc30 2021-03-09T09:40:18.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98404f440 con 0x7ff98001dc30 2021-03-09T09:40:18.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98404f440 con 0x7ff98001dc30 2021-03-09T09:40:18.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.988+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:18.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.988+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:18.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:18.988+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:23.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:23.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:23.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004aed0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff9840813b0 tx=0x7ff98402c120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:23.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004aed0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff9840813b0 tx=0x7ff98402c120).stop 2021-03-09T09:40:23.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004aed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:23.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004aed0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:23.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980038fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:23.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:23.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:23.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004e4e0 con 0x7ff99c0adc20 2021-03-09T09:40:23.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980031a00 con 0x7ff99c0613a0 2021-03-09T09:40:23.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ed40 con 0x7ff980087a10 2021-03-09T09:40:23.981 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:23.976+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 411 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:23.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:23.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:23.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:23.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:23.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980038fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:23.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:23.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:23.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:23.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:23.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980076300 con 0x7ff980087a10 2021-03-09T09:40:23.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9840096b0 con 0x7ff980087a10 2021-03-09T09:40:23.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980038fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff984009680 tx=0x7ff98400f510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:23.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019040 con 0x7ff980087a10 2021-03-09T09:40:23.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98405d000 con 0x7ff980087a10 2021-03-09T09:40:23.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984003b10 con 0x7ff980087a10 2021-03-09T09:40:23.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.988+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:23.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.988+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:23.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:23.988+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:24.514 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:40:24.517 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:40:28.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980038fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:28.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980038fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:28.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980038fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff984009680 tx=0x7ff98400f510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:28.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980038fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff984009680 tx=0x7ff98400f510).stop 2021-03-09T09:40:28.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980038fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:28.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980038fe0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:28.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:28.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:28.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:28.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039b50 con 0x7ff99c0adc20 2021-03-09T09:40:28.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980021760 con 0x7ff99c0613a0 2021-03-09T09:40:28.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004ec0 con 0x7ff98001dc30 2021-03-09T09:40:28.981 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:28.976+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 446 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:28.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:28.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:28.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:28.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:28.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:28.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:28.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:28.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:28.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:28.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001b390 con 0x7ff98001dc30 2021-03-09T09:40:28.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004e620 con 0x7ff98001dc30 2021-03-09T09:40:28.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7ff98c0a1d70 tx=0x7ff98c07a1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:28.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023040 con 0x7ff98001dc30 2021-03-09T09:40:28.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98001dc30 2021-03-09T09:40:28.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98001dc30 2021-03-09T09:40:28.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.984+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:28.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:28.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:28.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:33.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980095350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:33.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980095350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:33.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7ff98c0a1d70 tx=0x7ff98c07a1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:33.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7ff98c0a1d70 tx=0x7ff98c07a1d0).stop 2021-03-09T09:40:33.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980095350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:33.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095350 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:33.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98002cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:33.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:33.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:33.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038970 con 0x7ff99c0adc20 2021-03-09T09:40:33.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000e3b0 con 0x7ff99c0613a0 2021-03-09T09:40:33.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b240 con 0x7ff980087a10 2021-03-09T09:40:33.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:33.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:33.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:33.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:33.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98002cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:33.985 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:33.976+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 484 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:33.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:33.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:33.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:33.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:33.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800188f0 con 0x7ff980087a10 2021-03-09T09:40:33.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9840096b0 con 0x7ff980087a10 2021-03-09T09:40:33.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98002cec0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7ff98404bfe0 tx=0x7ff98404bcd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:33.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980087a10 2021-03-09T09:40:33.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98402c190 con 0x7ff980087a10 2021-03-09T09:40:33.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980087a10 2021-03-09T09:40:33.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.984+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:33.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:33.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:33.984+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:38.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98002cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:38.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98002cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:38.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98002cec0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7ff98404bfe0 tx=0x7ff98404bcd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:38.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98002cec0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7ff98404bfe0 tx=0x7ff98404bcd0).stop 2021-03-09T09:40:38.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98002cec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:38.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98002cec0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:38.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:38.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:38.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:38.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017ab0 con 0x7ff99c0adc20 2021-03-09T09:40:38.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004cca0 con 0x7ff99c0613a0 2021-03-09T09:40:38.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003e3f0 con 0x7ff98001dc30 2021-03-09T09:40:38.984 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:38.976+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 519 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:38.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:38.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:38.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:38.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.976+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:38.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:38.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:38.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:38.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:38.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:38.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff98001dc30 2021-03-09T09:40:38.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001b390 con 0x7ff98001dc30 2021-03-09T09:40:38.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800359e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7ff98c09c9c0 tx=0x7ff98c00bb80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:38.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023020 con 0x7ff98001dc30 2021-03-09T09:40:38.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff98001dc30 2021-03-09T09:40:38.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98001dc30 2021-03-09T09:40:38.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.988+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:38.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.988+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:38.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:38.988+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:43.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800359e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:43.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800359e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:43.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800359e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7ff98c09c9c0 tx=0x7ff98c00bb80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:43.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800359e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7ff98c09c9c0 tx=0x7ff98c00bb80).stop 2021-03-09T09:40:43.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800359e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:43.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800359e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:43.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:43.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:43.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:43.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029710 con 0x7ff99c0adc20 2021-03-09T09:40:43.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b0a0 con 0x7ff99c0613a0 2021-03-09T09:40:43.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027120 con 0x7ff980087a10 2021-03-09T09:40:43.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:43.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:43.985 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:43.980+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 556 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:43.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:43.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:43.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:43.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:43.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:43.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:43.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:43.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096440 con 0x7ff980087a10 2021-03-09T09:40:43.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff980087a10 2021-03-09T09:40:43.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095f50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff98c051510 tx=0x7ff98c051400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:43.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023070 con 0x7ff980087a10 2021-03-09T09:40:43.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff980087a10 2021-03-09T09:40:43.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c050d00 con 0x7ff980087a10 2021-03-09T09:40:43.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.992+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:43.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.992+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:43.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:43.992+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:48.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980095f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:48.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980095f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:48.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095f50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff98c051510 tx=0x7ff98c051400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:48.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095f50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff98c051510 tx=0x7ff98c051400).stop 2021-03-09T09:40:48.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980095f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:48.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095f50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:48.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:48.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:48.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:48.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800599e0 con 0x7ff99c0adc20 2021-03-09T09:40:48.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003e400 con 0x7ff99c0613a0 2021-03-09T09:40:48.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a470 con 0x7ff98001dc30 2021-03-09T09:40:48.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:48.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:48.986 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:48.980+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 591 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:48.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:48.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:48.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.980+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:48.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:48.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:48.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:48.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:48.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98005fd00 con 0x7ff98001dc30 2021-03-09T09:40:48.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000e4f0 con 0x7ff98001dc30 2021-03-09T09:40:48.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800965d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7ff98006bad0 tx=0x7ff994000f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:48.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff98001dc30 2021-03-09T09:40:48.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98001dc30 2021-03-09T09:40:48.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99405c990 con 0x7ff98001dc30 2021-03-09T09:40:48.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.996+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:48.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:48.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:48.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:53.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:53.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:53.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800965d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7ff98006bad0 tx=0x7ff994000f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:53.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800965d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7ff98006bad0 tx=0x7ff994000f80).stop 2021-03-09T09:40:53.984 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:53.980+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 627 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:53.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800965d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:53.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800965d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:53.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:53.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:53.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:53.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004c750 con 0x7ff99c0adc20 2021-03-09T09:40:53.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:53.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012db0 con 0x7ff99c0613a0 2021-03-09T09:40:53.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980070fa0 con 0x7ff980087a10 2021-03-09T09:40:53.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:53.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:53.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:53.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:53.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:53.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:53.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:53.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:53.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045fc0 con 0x7ff980087a10 2021-03-09T09:40:53.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096440 con 0x7ff980087a10 2021-03-09T09:40:53.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096ac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c0066c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:53.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023040 con 0x7ff980087a10 2021-03-09T09:40:53.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980087a10 2021-03-09T09:40:53.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c028c40 con 0x7ff980087a10 2021-03-09T09:40:53.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.988+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:53.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.992+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:53.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:53.992+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:40:54.564 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:40:54.567 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:40:58.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980096ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:58.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980096ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:58.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096ac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c0066c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:58.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096ac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c0066c0).stop 2021-03-09T09:40:58.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980096ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:58.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096ac0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:58.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:58.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:58.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:58.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980070fa0 con 0x7ff99c0adc20 2021-03-09T09:40:58.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004cca0 con 0x7ff99c0613a0 2021-03-09T09:40:58.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ffc0 con 0x7ff98001dc30 2021-03-09T09:40:58.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:40:58.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:58.988 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:40:58.980+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 664 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:40:58.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:40:58.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.980+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:58.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:58.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:58.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:58.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:58.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:58.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980059990 con 0x7ff98001dc30 2021-03-09T09:40:58.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98005fd00 con 0x7ff98001dc30 2021-03-09T09:40:58.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008e7b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7ff98003e590 tx=0x7ff994069fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:58.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a4b0 con 0x7ff98001dc30 2021-03-09T09:40:58.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400b9f0 con 0x7ff98001dc30 2021-03-09T09:40:58.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99405c990 con 0x7ff98001dc30 2021-03-09T09:40:58.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.992+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:58.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.992+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:58.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:40:58.992+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:03.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98008e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:03.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98008e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:03.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008e7b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7ff98003e590 tx=0x7ff994069fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:03.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008e7b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7ff98003e590 tx=0x7ff994069fa0).stop 2021-03-09T09:41:03.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98008e7b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:03.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008e7b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:03.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:03.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980022e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:03.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:03.987 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:03.984+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 701 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:03.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:03.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:03.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980070fa0 con 0x7ff99c0adc20 2021-03-09T09:41:03.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980062ba0 con 0x7ff99c0613a0 2021-03-09T09:41:03.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051110 con 0x7ff980087a10 2021-03-09T09:41:03.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:03.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:03.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980022e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:03.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:03.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:03.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:03.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:03.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000a320 con 0x7ff980087a10 2021-03-09T09:41:03.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045fc0 con 0x7ff980087a10 2021-03-09T09:41:03.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.988+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980022e50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c096c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:03.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023070 con 0x7ff980087a10 2021-03-09T09:41:03.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d020 con 0x7ff980087a10 2021-03-09T09:41:03.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980087a10 2021-03-09T09:41:03.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.992+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:03.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.992+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:03.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:03.992+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:08.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980022e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:08.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980022e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:08.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980022e50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c096c00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:08.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980022e50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c096c00).stop 2021-03-09T09:41:08.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980022e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:08.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980022e50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:08.987 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:08.984+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 734 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:08.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:08.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:08.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:08.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053830 con 0x7ff99c0adc20 2021-03-09T09:41:08.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980062ba0 con 0x7ff99c0613a0 2021-03-09T09:41:08.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001ba70 con 0x7ff98001dc30 2021-03-09T09:41:08.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:08.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:08.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:08.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:08.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:08.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:08.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:08.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:08.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:08.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800304b0 con 0x7ff98001dc30 2021-03-09T09:41:08.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000a320 con 0x7ff98001dc30 2021-03-09T09:41:08.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095cf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7ff98c096f30 tx=0x7ff98c0a6f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:08.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023070 con 0x7ff98001dc30 2021-03-09T09:41:08.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c096fa0 con 0x7ff98001dc30 2021-03-09T09:41:08.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04c8e0 con 0x7ff98001dc30 2021-03-09T09:41:08.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.992+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:08.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.992+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:08.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:08.992+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:13.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980095cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:13.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980095cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:13.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095cf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7ff98c096f30 tx=0x7ff98c0a6f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:13.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095cf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7ff98c096f30 tx=0x7ff98c0a6f40).stop 2021-03-09T09:41:13.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980095cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:13.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095cf0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:13.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980090780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:13.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:13.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:13.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033820 con 0x7ff99c0adc20 2021-03-09T09:41:13.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980070fa0 con 0x7ff99c0613a0 2021-03-09T09:41:13.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980062ba0 con 0x7ff980087a10 2021-03-09T09:41:13.989 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:13.984+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 771 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:13.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:13.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:13.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:13.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:13.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.984+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980090780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:13.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:13.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:13.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:13.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:13.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98005fd00 con 0x7ff980087a10 2021-03-09T09:41:13.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800188f0 con 0x7ff980087a10 2021-03-09T09:41:13.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980090780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7ff984049fd0 tx=0x7ff98407b460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:13.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019030 con 0x7ff980087a10 2021-03-09T09:41:13.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400f180 con 0x7ff980087a10 2021-03-09T09:41:13.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980087a10 2021-03-09T09:41:13.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.996+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:13.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:13.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:13.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:18.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980090780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:18.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980090780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:18.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980090780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7ff984049fd0 tx=0x7ff98407b460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:18.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980090780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7ff984049fd0 tx=0x7ff98407b460).stop 2021-03-09T09:41:18.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980090780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:18.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980090780 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:18.989 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:18.984+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 807 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:18.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:18.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:18.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800943b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:18.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c7c0 con 0x7ff99c0adc20 2021-03-09T09:41:18.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018700 con 0x7ff99c0613a0 2021-03-09T09:41:18.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c0f0 con 0x7ff98001dc30 2021-03-09T09:41:18.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:18.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:18.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:18.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:18.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800943b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:18.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:18.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:18.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:18.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:18.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980004d30 con 0x7ff98001dc30 2021-03-09T09:41:18.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980059990 con 0x7ff98001dc30 2021-03-09T09:41:18.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800943b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ff98006e430 tx=0x7ff994065e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:18.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405c990 con 0x7ff98001dc30 2021-03-09T09:41:18.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400b9f0 con 0x7ff98001dc30 2021-03-09T09:41:18.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99405c990 con 0x7ff98001dc30 2021-03-09T09:41:18.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.996+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:18.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:18.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:18.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:23.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800943b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:23.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800943b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:23.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800943b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ff98006e430 tx=0x7ff994065e40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:23.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800943b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ff98006e430 tx=0x7ff994065e40).stop 2021-03-09T09:41:23.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800943b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:23.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800943b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:23.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:23.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:23.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:23.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004c9b0 con 0x7ff99c0adc20 2021-03-09T09:41:23.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ec00 con 0x7ff99c0613a0 2021-03-09T09:41:23.991 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:23.984+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 844 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:23.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800914d0 con 0x7ff980087a10 2021-03-09T09:41:23.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.984+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:23.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.988+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:23.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:23.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:23.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:23.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:23.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:23.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:23.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:23.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980062ba0 con 0x7ff980087a10 2021-03-09T09:41:23.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98005fd00 con 0x7ff980087a10 2021-03-09T09:41:23.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800968b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7ff984026070 tx=0x7ff984026200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:23.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980087a10 2021-03-09T09:41:23.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980087a10 2021-03-09T09:41:23.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980087a10 2021-03-09T09:41:23.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.996+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:23.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:24.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:23.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:24.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:41:24.613 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:41:28.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:28.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:28.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800968b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7ff984026070 tx=0x7ff984026200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:28.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800968b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7ff984026070 tx=0x7ff984026200).stop 2021-03-09T09:41:28.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800968b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:28.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800968b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:28.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980088bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:28.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:28.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:28.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800438a0 con 0x7ff99c0adc20 2021-03-09T09:41:28.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980021620 con 0x7ff99c0613a0 2021-03-09T09:41:28.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f310 con 0x7ff98001dc30 2021-03-09T09:41:28.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:28.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:28.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:28.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:28.994 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:28.988+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 878 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:28.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980088bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:28.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:28.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:28.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:28.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:28.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98005fd00 con 0x7ff98001dc30 2021-03-09T09:41:28.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980062ba0 con 0x7ff98001dc30 2021-03-09T09:41:28.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980088bb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7ff984023860 tx=0x7ff98404be90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:28.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff98001dc30 2021-03-09T09:41:28.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:28.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98403ea50 con 0x7ff98001dc30 2021-03-09T09:41:29.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:29.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984025860 con 0x7ff98001dc30 2021-03-09T09:41:29.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:29.000+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:29.000+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:29.000+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:33.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980088bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:33.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980088bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:33.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980088bb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7ff984023860 tx=0x7ff98404be90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:33.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980088bb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7ff984023860 tx=0x7ff98404be90).stop 2021-03-09T09:41:33.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980088bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:33.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980088bb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:33.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:33.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:33.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:33.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff99c0adc20 2021-03-09T09:41:33.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800226b0 con 0x7ff99c0613a0 2021-03-09T09:41:33.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055520 con 0x7ff980087a10 2021-03-09T09:41:33.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:33.993 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:33.988+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 915 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:33.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:33.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:33.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:33.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:33.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:33.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:33.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:33.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:33.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98006b9c0 con 0x7ff980087a10 2021-03-09T09:41:33.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980004d30 con 0x7ff980087a10 2021-03-09T09:41:33.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.992+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff99407a510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:33.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980087a10 2021-03-09T09:41:33.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405c990 con 0x7ff980087a10 2021-03-09T09:41:33.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994081940 con 0x7ff980087a10 2021-03-09T09:41:33.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:33.996+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:34.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:34.000+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:34.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:34.000+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:38.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:38.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:38.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff99407a510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:38.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff99407a510).stop 2021-03-09T09:41:38.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:38.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:38.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:38.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:38.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980049e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:38.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008db00 con 0x7ff99c0adc20 2021-03-09T09:41:38.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c0f0 con 0x7ff99c0613a0 2021-03-09T09:41:38.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff98001dc30 2021-03-09T09:41:38.993 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:38.988+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 952 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:38.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:38.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:38.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:38.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:38.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980049e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:38.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.992+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:38.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.992+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:38.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.992+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:38.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.992+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:38.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.992+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800604f0 con 0x7ff98001dc30 2021-03-09T09:41:38.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.992+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98006b9c0 con 0x7ff98001dc30 2021-03-09T09:41:38.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.992+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980049e30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7ff994033df0 tx=0x7ff994076e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:38.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994005a40 con 0x7ff98001dc30 2021-03-09T09:41:38.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994005ba0 con 0x7ff98001dc30 2021-03-09T09:41:38.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994005a40 con 0x7ff98001dc30 2021-03-09T09:41:38.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.996+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:38.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:39.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:38.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:43.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980049e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:43.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980049e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:43.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980049e30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7ff994033df0 tx=0x7ff994076e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:43.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980049e30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7ff994033df0 tx=0x7ff994076e90).stop 2021-03-09T09:41:43.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980049e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:43.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980049e30 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:43.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:43.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:43.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:43.994 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:43.988+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 987 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:43.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98005c6a0 con 0x7ff99c0adc20 2021-03-09T09:41:43.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030330 con 0x7ff99c0613a0 2021-03-09T09:41:43.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff980087a10 2021-03-09T09:41:43.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:43.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:43.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:43.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:43.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.988+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:43.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:43.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:43.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:43.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:43.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800227f0 con 0x7ff980087a10 2021-03-09T09:41:43.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98005fd00 con 0x7ff980087a10 2021-03-09T09:41:43.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7ff984039c40 tx=0x7ff984036a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:43.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980087a10 2021-03-09T09:41:43.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984026030 con 0x7ff980087a10 2021-03-09T09:41:44.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840587e0 con 0x7ff980087a10 2021-03-09T09:41:44.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.996+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:44.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:44.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:43.996+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:48.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:48.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:48.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7ff984039c40 tx=0x7ff984036a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:48.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7ff984039c40 tx=0x7ff984036a50).stop 2021-03-09T09:41:48.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:48.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:48.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980020590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:48.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:48.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:48.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff99c0adc20 2021-03-09T09:41:48.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017810 con 0x7ff99c0613a0 2021-03-09T09:41:48.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98006cce0 con 0x7ff98001dc30 2021-03-09T09:41:48.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:48.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:48.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:48.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:48.996 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:48.992+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1023 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:48.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980020590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:48.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:48.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:48.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:48.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:49.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980057a00 con 0x7ff98001dc30 2021-03-09T09:41:49.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800227f0 con 0x7ff98001dc30 2021-03-09T09:41:49.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980020590 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7ff984058d70 tx=0x7ff984026190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:49.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019040 con 0x7ff98001dc30 2021-03-09T09:41:49.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:48.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840440e0 con 0x7ff98001dc30 2021-03-09T09:41:49.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:49.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984036f20 con 0x7ff98001dc30 2021-03-09T09:41:49.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:49.000+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:49.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:49.000+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:49.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:49.000+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:53.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980020590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:41:53.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980020590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:53.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980020590 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7ff984058d70 tx=0x7ff984026190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:53.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980020590 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7ff984058d70 tx=0x7ff984026190).stop 2021-03-09T09:41:53.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980020590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:53.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980020590 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:53.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:53.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:53.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:53.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98006a6f0 con 0x7ff99c0adc20 2021-03-09T09:41:53.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800914d0 con 0x7ff99c0613a0 2021-03-09T09:41:53.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff980087a10 2021-03-09T09:41:53.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:53.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:53.996 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:53.992+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1059 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:53.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:53.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:53.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:53.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:53.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:53.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:53.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:53.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980004d30 con 0x7ff980087a10 2021-03-09T09:41:53.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.992+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800304b0 con 0x7ff980087a10 2021-03-09T09:41:53.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:53.996+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7ff98003e7f0 tx=0x7ff98c00baf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:54.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:54.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023040 con 0x7ff980087a10 2021-03-09T09:41:54.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:54.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d020 con 0x7ff980087a10 2021-03-09T09:41:54.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:54.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980087a10 2021-03-09T09:41:54.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:54.000+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:54.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:54.000+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:54.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:54.000+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:41:54.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:41:54.660 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:41:58.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:58.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:58.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7ff98003e7f0 tx=0x7ff98c00baf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:58.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7ff98003e7f0 tx=0x7ff98c00baf0).stop 2021-03-09T09:41:58.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:58.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:58.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800145e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:58.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:58.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:58.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002bac0 con 0x7ff99c0adc20 2021-03-09T09:41:58.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff99c0613a0 2021-03-09T09:41:58.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003eac0 con 0x7ff98001dc30 2021-03-09T09:41:58.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:41:58.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:58.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:41:58.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:58.998 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:41:58.992+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1096 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:41:58.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800145e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:58.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:59.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:59.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:59.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:59.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98005ad30 con 0x7ff98001dc30 2021-03-09T09:41:59.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980057a00 con 0x7ff98001dc30 2021-03-09T09:41:59.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:58.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800145e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98401d700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:59.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:59.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019020 con 0x7ff98001dc30 2021-03-09T09:41:59.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:59.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984046020 con 0x7ff98001dc30 2021-03-09T09:41:59.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:59.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff98001dc30 2021-03-09T09:41:59.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:59.004+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:59.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:59.004+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:59.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:41:59.004+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:03.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800145e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:03.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800145e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:03.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800145e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98401d700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:03.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800145e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98401d700).stop 2021-03-09T09:42:03.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800145e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:03.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800145e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:03.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:03.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:03.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:03.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff99c0adc20 2021-03-09T09:42:03.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039a10 con 0x7ff99c0613a0 2021-03-09T09:42:03.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000be30 con 0x7ff980087a10 2021-03-09T09:42:03.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:03.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:03.999 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:03.992+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1133 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:04.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:04.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:04.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.992+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:04.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:04.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:04.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:04.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:04.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980033770 con 0x7ff980087a10 2021-03-09T09:42:04.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98005ad30 con 0x7ff980087a10 2021-03-09T09:42:04.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:03.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7ff98404bed0 tx=0x7ff984026030).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:04.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:04.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019040 con 0x7ff980087a10 2021-03-09T09:42:04.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:04.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840ab0a0 con 0x7ff980087a10 2021-03-09T09:42:04.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:04.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980087a10 2021-03-09T09:42:04.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:04.004+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:04.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:04.008+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:04.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:04.008+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:08.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:08.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:08.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7ff98404bed0 tx=0x7ff984026030).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:08.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7ff98404bed0 tx=0x7ff984026030).stop 2021-03-09T09:42:08.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:08.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:08.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98003fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:08.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:08.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:08.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff99c0adc20 2021-03-09T09:42:08.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f310 con 0x7ff99c0613a0 2021-03-09T09:42:08.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b850 con 0x7ff98001dc30 2021-03-09T09:42:08.999 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:08.996+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1168 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:09.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:09.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:09.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:09.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:09.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:08.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98003fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:09.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.000+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:09.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.000+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:09.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.000+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:09.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.000+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:09.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.000+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98006cce0 con 0x7ff98001dc30 2021-03-09T09:42:09.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.000+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980033770 con 0x7ff98001dc30 2021-03-09T09:42:09.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.000+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98003fde0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840204c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:09.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff98001dc30 2021-03-09T09:42:09.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984028d60 con 0x7ff98001dc30 2021-03-09T09:42:09.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984036f20 con 0x7ff98001dc30 2021-03-09T09:42:09.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.008+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:09.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.008+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:09.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:09.008+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:13.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98003fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:13.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98003fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:13.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98003fde0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840204c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:13.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98003fde0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840204c0).stop 2021-03-09T09:42:13.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98003fde0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:13.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98003fde0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:13.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:13.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:13.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:14.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980031aa0 con 0x7ff99c0adc20 2021-03-09T09:42:14.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039a10 con 0x7ff99c0613a0 2021-03-09T09:42:14.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff980087a10 2021-03-09T09:42:14.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:14.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:14.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:14.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:14.002 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:13.996+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1210 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:14.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:13.996+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:14.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:14.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:14.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:14.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:14.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003bef0 con 0x7ff980087a10 2021-03-09T09:42:14.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800604f0 con 0x7ff980087a10 2021-03-09T09:42:14.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7ff980038b90 tx=0x7ff994081480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:14.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99401cac0 con 0x7ff980087a10 2021-03-09T09:42:14.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400b9f0 con 0x7ff980087a10 2021-03-09T09:42:14.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980087a10 2021-03-09T09:42:14.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.004+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:14.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.004+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:14.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:14.004+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:18.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:18.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:18.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7ff980038b90 tx=0x7ff994081480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:18.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7ff980038b90 tx=0x7ff994081480).stop 2021-03-09T09:42:18.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800963a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:18.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800963a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:18.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:19.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:19.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:19.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b850 con 0x7ff99c0adc20 2021-03-09T09:42:19.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff99c0613a0 2021-03-09T09:42:19.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038b90 con 0x7ff98001dc30 2021-03-09T09:42:19.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:19.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:19.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:19.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:19.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:18.996+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:19.003 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:18.996+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1239 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:19.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:19.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:19.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:19.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:19.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d660 con 0x7ff98001dc30 2021-03-09T09:42:19.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980004d30 con 0x7ff98001dc30 2021-03-09T09:42:19.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800988c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7ff9800906a0 tx=0x7ff98c04b8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:19.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023040 con 0x7ff98001dc30 2021-03-09T09:42:19.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff98001dc30 2021-03-09T09:42:19.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0203a0 con 0x7ff98001dc30 2021-03-09T09:42:19.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.012+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:19.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:19.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:19.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:23.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:23.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:23.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800988c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7ff9800906a0 tx=0x7ff98c04b8d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:23.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800988c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7ff9800906a0 tx=0x7ff98c04b8d0).stop 2021-03-09T09:42:24.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800988c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:24.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800988c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:24.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:24.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:24.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:24.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800497f0 con 0x7ff99c0adc20 2021-03-09T09:42:24.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff99c0613a0 2021-03-09T09:42:24.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800900d0 con 0x7ff980087a10 2021-03-09T09:42:24.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:24.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:24.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:24.002 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:23.996+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1274 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:24.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:24.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:23.996+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:24.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:24.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:24.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:24.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:24.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001f000 con 0x7ff980087a10 2021-03-09T09:42:24.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9940ade70 con 0x7ff980087a10 2021-03-09T09:42:24.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096380 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406ca70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:24.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405cba0 con 0x7ff980087a10 2021-03-09T09:42:24.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a650 con 0x7ff980087a10 2021-03-09T09:42:24.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940740e0 con 0x7ff980087a10 2021-03-09T09:42:24.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.008+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:24.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.008+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:24.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:24.008+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:24.712 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:42:24.715 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:42:29.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:28.996+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980096380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:42:29.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:28.996+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980096380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:28.996+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096380 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406ca70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:28.996+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096380 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406ca70).stop 2021-03-09T09:42:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:28.996+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980096380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:28.996+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980096380 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:29.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:29.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:29.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:29.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019220 con 0x7ff99c0adc20 2021-03-09T09:42:29.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:29.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800636d0 con 0x7ff99c0613a0 2021-03-09T09:42:29.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff98001dc30 2021-03-09T09:42:29.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:29.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:29.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:29.004 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:29.000+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1312 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:29.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:29.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:29.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:29.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:29.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:29.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980018b60 con 0x7ff98001dc30 2021-03-09T09:42:29.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d660 con 0x7ff98001dc30 2021-03-09T09:42:29.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.004+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098f00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7ff98c032ad0 tx=0x7ff98c0326e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:29.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023070 con 0x7ff98001dc30 2021-03-09T09:42:29.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff98001dc30 2021-03-09T09:42:29.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98001dc30 2021-03-09T09:42:29.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.012+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:29.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:29.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:29.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:34.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980098f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:34.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980098f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:34.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098f00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7ff98c032ad0 tx=0x7ff98c0326e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:34.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098f00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7ff98c032ad0 tx=0x7ff98c0326e0).stop 2021-03-09T09:42:34.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980098f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:34.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098f00 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:34.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:34.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:34.003 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:34.000+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1346 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:34.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:34.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f310 con 0x7ff99c0adc20 2021-03-09T09:42:34.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a070 con 0x7ff99c0613a0 2021-03-09T09:42:34.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b850 con 0x7ff980087a10 2021-03-09T09:42:34.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:34.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:34.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:34.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:34.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.000+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:34.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:34.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:34.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:34.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:34.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045410 con 0x7ff980087a10 2021-03-09T09:42:34.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98006cce0 con 0x7ff980087a10 2021-03-09T09:42:34.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800993f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7ff9840a9270 tx=0x7ff984006c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:34.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019030 con 0x7ff980087a10 2021-03-09T09:42:34.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff980087a10 2021-03-09T09:42:34.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff980087a10 2021-03-09T09:42:34.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.012+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:34.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:34.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:34.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:39.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:39.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:39.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800993f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7ff9840a9270 tx=0x7ff984006c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:39.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800993f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7ff9840a9270 tx=0x7ff984006c60).stop 2021-03-09T09:42:39.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800993f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:39.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800993f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:39.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:39.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:39.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:39.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:39.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:39.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039a10 con 0x7ff99c0adc20 2021-03-09T09:42:39.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ad20 con 0x7ff99c0613a0 2021-03-09T09:42:39.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023b60 con 0x7ff98001dc30 2021-03-09T09:42:39.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:39.006 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:39.000+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1382 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:39.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:39.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:39.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.004+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:39.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.004+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:39.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.004+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:39.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.004+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:39.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.004+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008dab0 con 0x7ff98001dc30 2021-03-09T09:42:39.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.004+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980018b60 con 0x7ff98001dc30 2021-03-09T09:42:39.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.004+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098e00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c00a230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:39.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff98001dc30 2021-03-09T09:42:39.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98001dc30 2021-03-09T09:42:39.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98001dc30 2021-03-09T09:42:39.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.008+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:39.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:39.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:39.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:44.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980098e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:44.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980098e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:44.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098e00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c00a230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:44.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098e00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c00a230).stop 2021-03-09T09:42:44.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980098e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:44.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980098e00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:44.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:44.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:44.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:44.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018460 con 0x7ff99c0adc20 2021-03-09T09:42:44.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0613a0 2021-03-09T09:42:44.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d8b0 con 0x7ff980087a10 2021-03-09T09:42:44.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:44.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:44.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:44.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.000+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:44.007 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:44.000+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1418 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:44.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:44.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.004+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:44.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:44.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.004+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:44.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:44.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.004+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800494f0 con 0x7ff980087a10 2021-03-09T09:42:44.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.004+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001f000 con 0x7ff980087a10 2021-03-09T09:42:44.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001a350 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99405cc80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:44.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980087a10 2021-03-09T09:42:44.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980087a10 2021-03-09T09:42:44.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99407a4b0 con 0x7ff980087a10 2021-03-09T09:42:44.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.016+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:44.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.016+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:44.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:44.016+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:49.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98001a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:42:49.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98001a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:49.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001a350 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99405cc80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:49.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001a350 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99405cc80).stop 2021-03-09T09:42:49.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98001a350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:49.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001a350 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:49.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:49.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:49.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:49.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022840 con 0x7ff99c0adc20 2021-03-09T09:42:49.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019220 con 0x7ff99c0613a0 2021-03-09T09:42:49.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.000+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b850 con 0x7ff98001dc30 2021-03-09T09:42:49.007 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:49.000+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1454 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:49.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:49.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:49.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:49.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:49.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:49.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:49.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:49.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:49.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:49.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a230 con 0x7ff98001dc30 2021-03-09T09:42:49.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045410 con 0x7ff98001dc30 2021-03-09T09:42:49.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004ee00 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984034190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:49.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff98001dc30 2021-03-09T09:42:49.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff98001dc30 2021-03-09T09:42:49.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff98001dc30 2021-03-09T09:42:49.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.012+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:49.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:49.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:49.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:54.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:54.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:54.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004ee00 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984034190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:54.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004ee00 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984034190).stop 2021-03-09T09:42:54.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004ee00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:54.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004ee00 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:54.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:54.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:54.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:54.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0adc20 2021-03-09T09:42:54.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980028b30 con 0x7ff99c0613a0 2021-03-09T09:42:54.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c770 con 0x7ff980087a10 2021-03-09T09:42:54.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:54.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:54.008 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:54.004+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1490 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:54.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:54.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:54.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:54.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:54.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:54.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:54.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:54.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980012d00 con 0x7ff980087a10 2021-03-09T09:42:54.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003a230 con 0x7ff980087a10 2021-03-09T09:42:54.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095840 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7ff9840342e0 tx=0x7ff9840258a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:54.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980087a10 2021-03-09T09:42:54.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff980087a10 2021-03-09T09:42:54.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980087a10 2021-03-09T09:42:54.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.012+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:54.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:54.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:54.012+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:42:54.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:42:54.777 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:42:59.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980095840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:59.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980095840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:59.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095840 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7ff9840342e0 tx=0x7ff9840258a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:59.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095840 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7ff9840342e0 tx=0x7ff9840258a0).stop 2021-03-09T09:42:59.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980095840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:59.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095840 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:59.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:59.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:59.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:59.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003e480 con 0x7ff99c0adc20 2021-03-09T09:42:59.009 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:42:59.004+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1525 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:42:59.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022a40 con 0x7ff99c0613a0 2021-03-09T09:42:59.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff98001dc30 2021-03-09T09:42:59.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:42:59.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:59.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:42:59.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:59.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:59.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:59.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:59.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:59.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:59.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002a120 con 0x7ff98001dc30 2021-03-09T09:42:59.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800494f0 con 0x7ff98001dc30 2021-03-09T09:42:59.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095d30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7ff994032010 tx=0x7ff99407af10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:59.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff98001dc30 2021-03-09T09:42:59.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98001dc30 2021-03-09T09:42:59.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99406ccd0 con 0x7ff98001dc30 2021-03-09T09:42:59.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.020+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:59.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:59.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:42:59.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:04.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980095d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:04.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980095d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:04.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095d30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7ff994032010 tx=0x7ff99407af10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:04.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095d30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7ff994032010 tx=0x7ff99407af10).stop 2021-03-09T09:43:04.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980095d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:04.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980095d30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:04.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:04.008 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:04.004+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1563 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:04.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:04.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:04.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0adc20 2021-03-09T09:43:04.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037d50 con 0x7ff99c0613a0 2021-03-09T09:43:04.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027860 con 0x7ff980087a10 2021-03-09T09:43:04.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:04.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:04.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:04.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.004+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:04.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:04.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:04.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:04.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:04.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:04.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003f6b0 con 0x7ff980087a10 2021-03-09T09:43:04.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002a120 con 0x7ff980087a10 2021-03-09T09:43:04.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099270 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7ff994077c40 tx=0x7ff9940ae6e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:04.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980087a10 2021-03-09T09:43:04.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081340 con 0x7ff980087a10 2021-03-09T09:43:04.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99407a4b0 con 0x7ff980087a10 2021-03-09T09:43:04.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.016+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:04.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.016+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:04.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:04.016+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:09.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980099270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:09.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980099270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:09.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099270 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7ff994077c40 tx=0x7ff9940ae6e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:09.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099270 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7ff994077c40 tx=0x7ff9940ae6e0).stop 2021-03-09T09:43:09.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980099270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:09.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099270 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:09.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:09.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:09.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:09.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980074510 con 0x7ff99c0adc20 2021-03-09T09:43:09.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a8d0 con 0x7ff99c0613a0 2021-03-09T09:43:09.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800902d0 con 0x7ff98001dc30 2021-03-09T09:43:09.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:09.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.004+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:09.010 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:09.004+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1598 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:09.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:09.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:09.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:09.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:09.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:09.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:09.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:09.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002a120 con 0x7ff98001dc30 2021-03-09T09:43:09.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980012d00 con 0x7ff98001dc30 2021-03-09T09:43:09.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000b220 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7ff984031900 tx=0x7ff9840a03d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:09.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019080 con 0x7ff98001dc30 2021-03-09T09:43:09.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840a20d0 con 0x7ff98001dc30 2021-03-09T09:43:09.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400f180 con 0x7ff98001dc30 2021-03-09T09:43:09.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.020+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:09.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:09.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:09.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:14.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.004+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98000b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:14.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98000b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:14.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000b220 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7ff984031900 tx=0x7ff9840a03d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:14.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000b220 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7ff984031900 tx=0x7ff9840a03d0).stop 2021-03-09T09:43:14.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98000b220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:14.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000b220 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:14.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:14.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:14.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:14.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0adc20 2021-03-09T09:43:14.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045460 con 0x7ff99c0613a0 2021-03-09T09:43:14.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003af20 con 0x7ff980087a10 2021-03-09T09:43:14.011 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:14.008+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1634 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:14.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:14.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:14.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:14.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:14.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:14.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:14.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:14.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:14.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:14.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004f580 con 0x7ff980087a10 2021-03-09T09:43:14.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002a120 con 0x7ff980087a10 2021-03-09T09:43:14.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099120 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7ff9840a2040 tx=0x7ff984009120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:14.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019040 con 0x7ff980087a10 2021-03-09T09:43:14.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984031660 con 0x7ff980087a10 2021-03-09T09:43:14.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff980087a10 2021-03-09T09:43:14.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.020+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:14.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:14.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:14.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:19.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980099120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:43:19.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980099120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:19.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099120 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7ff9840a2040 tx=0x7ff984009120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:19.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099120 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7ff9840a2040 tx=0x7ff984009120).stop 2021-03-09T09:43:19.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980099120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:19.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099120 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:19.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:19.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:19.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:19.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002bb40 con 0x7ff99c0adc20 2021-03-09T09:43:19.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093f00 con 0x7ff99c0613a0 2021-03-09T09:43:19.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff98001dc30 2021-03-09T09:43:19.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:19.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:19.015 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:19.008+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1670 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:19.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:19.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:19.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:19.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:19.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:19.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:19.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:19.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008d920 con 0x7ff98001dc30 2021-03-09T09:43:19.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f6b0 con 0x7ff98001dc30 2021-03-09T09:43:19.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056e50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940397b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:19.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994077a70 con 0x7ff98001dc30 2021-03-09T09:43:19.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a4b0 con 0x7ff98001dc30 2021-03-09T09:43:19.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99406d1f0 con 0x7ff98001dc30 2021-03-09T09:43:19.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.020+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:19.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:19.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:19.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:24.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980056e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:24.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980056e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:24.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056e50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940397b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:24.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056e50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940397b0).stop 2021-03-09T09:43:24.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980056e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:24.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056e50 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:24.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:24.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:24.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:24.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800494b0 con 0x7ff99c0adc20 2021-03-09T09:43:24.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038740 con 0x7ff99c0613a0 2021-03-09T09:43:24.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093f00 con 0x7ff980087a10 2021-03-09T09:43:24.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:24.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:24.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:24.013 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:24.008+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1707 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:24.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:24.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:24.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.012+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:24.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:24.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.012+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:24.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:24.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.012+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980041df0 con 0x7ff980087a10 2021-03-09T09:43:24.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.012+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008d920 con 0x7ff980087a10 2021-03-09T09:43:24.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7ff980020220 tx=0x7ff994028320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:24.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980087a10 2021-03-09T09:43:24.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994039330 con 0x7ff980087a10 2021-03-09T09:43:24.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940394a0 con 0x7ff980087a10 2021-03-09T09:43:24.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.020+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:24.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:24.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:24.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:24.920 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:43:24.923 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:43:29.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:29.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:29.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7ff980020220 tx=0x7ff994028320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:29.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7ff980020220 tx=0x7ff994028320).stop 2021-03-09T09:43:29.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800982b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:29.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:29.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:29.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:29.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:29.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:29.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012bc0 con 0x7ff99c0adc20 2021-03-09T09:43:29.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.008+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:29.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c190 con 0x7ff99c0613a0 2021-03-09T09:43:29.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f700 con 0x7ff98001dc30 2021-03-09T09:43:29.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:29.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:29.015 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:29.012+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1743 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:29.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:29.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:29.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:29.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:29.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:29.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013f80 con 0x7ff98001dc30 2021-03-09T09:43:29.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008dab0 con 0x7ff98001dc30 2021-03-09T09:43:29.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.016+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056200 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7ff980088750 tx=0x7ff98c00b720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:29.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff98001dc30 2021-03-09T09:43:29.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98001dc30 2021-03-09T09:43:29.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98001dc30 2021-03-09T09:43:29.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.020+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:29.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:29.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:29.020+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:34.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980056200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:34.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980056200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:34.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056200 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7ff980088750 tx=0x7ff98c00b720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:34.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056200 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7ff980088750 tx=0x7ff98c00b720).stop 2021-03-09T09:43:34.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980056200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:34.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980056200 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:34.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:34.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:34.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:34.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002bb40 con 0x7ff99c0adc20 2021-03-09T09:43:34.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0613a0 2021-03-09T09:43:34.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f700 con 0x7ff980087a10 2021-03-09T09:43:34.015 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:34.012+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1778 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:34.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:34.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:34.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:34.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:34.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:34.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:34.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:34.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:34.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:34.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001cea0 con 0x7ff980087a10 2021-03-09T09:43:34.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.012+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004f580 con 0x7ff980087a10 2021-03-09T09:43:34.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff984025950 tx=0x7ff9840a0110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:34.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984031b10 con 0x7ff980087a10 2021-03-09T09:43:34.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984031b10 con 0x7ff980087a10 2021-03-09T09:43:34.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980087a10 2021-03-09T09:43:34.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.024+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:34.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:34.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:34.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:39.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:39.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:39.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff984025950 tx=0x7ff9840a0110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:39.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff984025950 tx=0x7ff9840a0110).stop 2021-03-09T09:43:39.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800982b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:39.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800982b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:39.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:39.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:39.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:39.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980031a80 con 0x7ff99c0adc20 2021-03-09T09:43:39.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004db60 con 0x7ff99c0613a0 2021-03-09T09:43:39.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002bd40 con 0x7ff98001dc30 2021-03-09T09:43:39.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:39.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:39.017 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:39.012+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1814 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:39.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:39.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:39.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.012+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:39.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:39.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:39.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:39.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:39.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000e9f0 con 0x7ff98001dc30 2021-03-09T09:43:39.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001cea0 con 0x7ff98001dc30 2021-03-09T09:43:39.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840273a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:39.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019090 con 0x7ff98001dc30 2021-03-09T09:43:39.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840a2070 con 0x7ff98001dc30 2021-03-09T09:43:39.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984046280 con 0x7ff98001dc30 2021-03-09T09:43:39.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.024+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:39.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:39.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:39.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:44.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98000e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:44.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98000e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:44.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840273a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:44.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840273a0).stop 2021-03-09T09:43:44.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98000e500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:44.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98000e500 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:44.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:44.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:44.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:44.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092f00 con 0x7ff99c0adc20 2021-03-09T09:43:44.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0613a0 2021-03-09T09:43:44.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052330 con 0x7ff980087a10 2021-03-09T09:43:44.017 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:44.012+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1853 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:44.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:44.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:44.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:44.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:44.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:44.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.016+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:44.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.016+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:44.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.016+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:44.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.016+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:44.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.016+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980028790 con 0x7ff980087a10 2021-03-09T09:43:44.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.016+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013f80 con 0x7ff980087a10 2021-03-09T09:43:44.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098110 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7ff98005ddc0 tx=0x7ff98c00be80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:44.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff980087a10 2021-03-09T09:43:44.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980087a10 2021-03-09T09:43:44.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980087a10 2021-03-09T09:43:44.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.024+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:44.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:44.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:44.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:49.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980098110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:49.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980098110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:49.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098110 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7ff98005ddc0 tx=0x7ff98c00be80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:49.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098110 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7ff98005ddc0 tx=0x7ff98c00be80).stop 2021-03-09T09:43:49.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980098110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:49.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098110 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:49.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:49.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:49.017 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:49.016+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1888 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:49.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:49.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800164a0 con 0x7ff99c0adc20 2021-03-09T09:43:49.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a540 con 0x7ff99c0613a0 2021-03-09T09:43:49.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f700 con 0x7ff98001dc30 2021-03-09T09:43:49.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:49.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:49.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:49.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:49.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:49.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:49.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:49.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:49.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:49.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980036ef0 con 0x7ff98001dc30 2021-03-09T09:43:49.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980041df0 con 0x7ff98001dc30 2021-03-09T09:43:49.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008d130 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994065e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:49.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99401cb90 con 0x7ff98001dc30 2021-03-09T09:43:49.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940779a0 con 0x7ff98001dc30 2021-03-09T09:43:49.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98001dc30 2021-03-09T09:43:49.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.024+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:49.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:49.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:49.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:54.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98008d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:54.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98008d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:54.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008d130 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994065e40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:54.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008d130 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994065e40).stop 2021-03-09T09:43:54.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98008d130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:54.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008d130 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:54.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:54.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:54.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:54.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090170 con 0x7ff99c0adc20 2021-03-09T09:43:54.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002bd40 con 0x7ff99c0613a0 2021-03-09T09:43:54.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020110 con 0x7ff980087a10 2021-03-09T09:43:54.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:54.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:54.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:54.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:54.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:54.020 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:54.016+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1921 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:54.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:54.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:54.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:54.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:54.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a600 con 0x7ff980087a10 2021-03-09T09:43:54.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000e9f0 con 0x7ff980087a10 2021-03-09T09:43:54.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097bd0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7ff98400f460 tx=0x7ff98402ca10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:54.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984027610 con 0x7ff980087a10 2021-03-09T09:43:54.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98402fd90 con 0x7ff980087a10 2021-03-09T09:43:54.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400f180 con 0x7ff980087a10 2021-03-09T09:43:54.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.024+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:54.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:54.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:54.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:43:54.957 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:43:54.963 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:43:59.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980097bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:59.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980097bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:59.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097bd0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7ff98400f460 tx=0x7ff98402ca10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:59.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097bd0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7ff98400f460 tx=0x7ff98402ca10).stop 2021-03-09T09:43:59.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980097bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:59.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097bd0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:59.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980011a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:59.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:59.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:59.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0adc20 2021-03-09T09:43:59.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980028650 con 0x7ff99c0613a0 2021-03-09T09:43:59.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800579a0 con 0x7ff98001dc30 2021-03-09T09:43:59.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:43:59.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:43:59.020 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:43:59.016+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1958 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:43:59.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:59.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:59.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980011a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:59.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.020+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:59.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.020+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:59.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.020+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:59.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.020+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:59.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.020+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002a120 con 0x7ff98001dc30 2021-03-09T09:43:59.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.020+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003a600 con 0x7ff98001dc30 2021-03-09T09:43:59.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.020+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980011a60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840583f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:59.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019080 con 0x7ff98001dc30 2021-03-09T09:43:59.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840587e0 con 0x7ff98001dc30 2021-03-09T09:43:59.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984022e10 con 0x7ff98001dc30 2021-03-09T09:43:59.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.028+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:59.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.028+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:59.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:43:59.028+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:04.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980011a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:44:04.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980011a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:04.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980011a60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840583f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:04.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980011a60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840583f0).stop 2021-03-09T09:44:04.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980011a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:04.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980011a60 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:04.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:04.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980031280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:04.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:04.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0adc20 2021-03-09T09:44:04.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800118d0 con 0x7ff99c0613a0 2021-03-09T09:44:04.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037d50 con 0x7ff980087a10 2021-03-09T09:44:04.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:04.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:04.024 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:04.016+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1993 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:04.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:04.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:04.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980031280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:04.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:04.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:04.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:04.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:04.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980089d10 con 0x7ff980087a10 2021-03-09T09:44:04.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980028790 con 0x7ff980087a10 2021-03-09T09:44:04.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980031280 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c09d070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:04.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff980087a10 2021-03-09T09:44:04.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d020 con 0x7ff980087a10 2021-03-09T09:44:04.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980087a10 2021-03-09T09:44:04.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.028+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:04.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.028+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:04.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:04.028+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:09.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980031280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:09.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980031280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:09.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980031280 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c09d070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:09.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980031280 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c09d070).stop 2021-03-09T09:44:09.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980031280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:09.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980031280 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:09.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980012820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:09.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:09.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:09.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0adc20 2021-03-09T09:44:09.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800141d0 con 0x7ff99c0613a0 2021-03-09T09:44:09.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800951c0 con 0x7ff98001dc30 2021-03-09T09:44:09.022 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:09.020+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2030 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:09.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:09.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:09.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:09.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:09.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.020+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980012820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:09.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:09.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:09.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:09.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:09.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000f910 con 0x7ff98001dc30 2021-03-09T09:44:09.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002a120 con 0x7ff98001dc30 2021-03-09T09:44:09.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980012820 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984041510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:09.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019040 con 0x7ff98001dc30 2021-03-09T09:44:09.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984034180 con 0x7ff98001dc30 2021-03-09T09:44:09.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840271a0 con 0x7ff98001dc30 2021-03-09T09:44:09.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.032+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:09.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.032+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:09.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:09.032+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:14.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980012820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:14.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980012820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:14.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980012820 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984041510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:14.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980012820 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984041510).stop 2021-03-09T09:44:14.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980012820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:14.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980012820 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:14.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:14.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980028980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:14.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:14.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800141d0 con 0x7ff99c0adc20 2021-03-09T09:44:14.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050d80 con 0x7ff99c0613a0 2021-03-09T09:44:14.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800266b0 con 0x7ff980087a10 2021-03-09T09:44:14.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:14.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:14.026 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:14.020+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2065 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:14.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:14.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:14.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980028980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:14.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:14.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:14.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:14.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:14.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980039900 con 0x7ff980087a10 2021-03-09T09:44:14.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980089d10 con 0x7ff980087a10 2021-03-09T09:44:14.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.024+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980028980 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7ff980055ff0 tx=0x7ff98c0048a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:14.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053030 con 0x7ff980087a10 2021-03-09T09:44:14.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff980087a10 2021-03-09T09:44:14.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c004d00 con 0x7ff980087a10 2021-03-09T09:44:14.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.032+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:14.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.032+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:14.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:14.032+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:19.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980028980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:19.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980028980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:19.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980028980 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7ff980055ff0 tx=0x7ff98c0048a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:19.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980028980 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7ff980055ff0 tx=0x7ff98c0048a0).stop 2021-03-09T09:44:19.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980028980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:19.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980028980 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:19.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:19.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:19.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980044820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:19.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800389a0 con 0x7ff99c0adc20 2021-03-09T09:44:19.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d850 con 0x7ff99c0613a0 2021-03-09T09:44:19.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cb80 con 0x7ff98001dc30 2021-03-09T09:44:19.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:19.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:19.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:19.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:19.025 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:19.020+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2102 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:19.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980044820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:19.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:19.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:19.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:19.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98008a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:19.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003b410 con 0x7ff98001dc30 2021-03-09T09:44:19.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980036ef0 con 0x7ff98001dc30 2021-03-09T09:44:19.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980044820 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7ff98002bca0 tx=0x7ff994077d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:19.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994036030 con 0x7ff98001dc30 2021-03-09T09:44:19.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994035e00 con 0x7ff98001dc30 2021-03-09T09:44:19.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940791f0 con 0x7ff98001dc30 2021-03-09T09:44:19.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.032+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:19.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.032+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:19.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:19.032+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:24.022 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:24.020+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2140 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:24.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980044820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:24.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980044820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:24.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980044820 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7ff98002bca0 tx=0x7ff994077d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:24.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980044820 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7ff98002bca0 tx=0x7ff994077d60).stop 2021-03-09T09:44:24.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980044820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:24.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980044820 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:24.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:24.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:24.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98008a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:24.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045660 con 0x7ff99c0adc20 2021-03-09T09:44:24.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0613a0 2021-03-09T09:44:24.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:24.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800121e0 con 0x7ff980087a10 2021-03-09T09:44:24.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:24.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:24.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:24.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98008a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:24.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:24.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:24.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:24.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:24.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093d50 con 0x7ff980087a10 2021-03-09T09:44:24.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003b410 con 0x7ff980087a10 2021-03-09T09:44:24.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98008a940 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994069ce0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:24.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980087a10 2021-03-09T09:44:24.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994028030 con 0x7ff980087a10 2021-03-09T09:44:24.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994028030 con 0x7ff980087a10 2021-03-09T09:44:24.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.032+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:24.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.032+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:24.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:24.032+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:25.004 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:44:25.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:44:29.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98008a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:29.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98008a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:29.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98008a940 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994069ce0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:29.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98008a940 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994069ce0).stop 2021-03-09T09:44:29.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98008a940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:29.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98008a940 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:29.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:29.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:29.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:29.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030bf0 con 0x7ff99c0adc20 2021-03-09T09:44:29.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d630 con 0x7ff99c0613a0 2021-03-09T09:44:29.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037f50 con 0x7ff98001dc30 2021-03-09T09:44:29.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:29.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:29.026 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:29.024+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2176 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:29.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:29.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:29.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:29.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:29.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:29.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:29.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:29.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980008210 con 0x7ff98001dc30 2021-03-09T09:44:29.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000f910 con 0x7ff98001dc30 2021-03-09T09:44:29.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008ba60 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840412d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:29.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff98001dc30 2021-03-09T09:44:29.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984034180 con 0x7ff98001dc30 2021-03-09T09:44:29.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff98001dc30 2021-03-09T09:44:29.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.036+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:29.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:29.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:29.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:34.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98008ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:34.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98008ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:34.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008ba60 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840412d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:34.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008ba60 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840412d0).stop 2021-03-09T09:44:34.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98008ba60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:34.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98008ba60 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:34.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:34.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:34.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:34.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a540 con 0x7ff99c0adc20 2021-03-09T09:44:34.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037d50 con 0x7ff99c0613a0 2021-03-09T09:44:34.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800092b0 con 0x7ff980087a10 2021-03-09T09:44:34.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:34.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:34.028 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:34.024+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2212 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:34.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:34.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:34.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:34.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:34.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:34.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:34.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:34.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008ebb0 con 0x7ff980087a10 2021-03-09T09:44:34.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980008210 con 0x7ff980087a10 2021-03-09T09:44:34.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840468a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:34.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980087a10 2021-03-09T09:44:34.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984031b10 con 0x7ff980087a10 2021-03-09T09:44:34.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980087a10 2021-03-09T09:44:34.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.032+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:34.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:34.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:34.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:39.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:39.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:39.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840468a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:39.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840468a0).stop 2021-03-09T09:44:39.028 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:39.024+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2246 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:39.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980005520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:39.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:39.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:39.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:39.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:39.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0adc20 2021-03-09T09:44:39.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004bb30 con 0x7ff99c0613a0 2021-03-09T09:44:39.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043cf0 con 0x7ff98001dc30 2021-03-09T09:44:39.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:39.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:39.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:39.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.024+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:39.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:39.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:39.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:39.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:39.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:39.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050ae0 con 0x7ff98001dc30 2021-03-09T09:44:39.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ebb0 con 0x7ff98001dc30 2021-03-09T09:44:39.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099820 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7ff9840469b0 tx=0x7ff98402d3c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:39.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019020 con 0x7ff98001dc30 2021-03-09T09:44:39.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984061630 con 0x7ff98001dc30 2021-03-09T09:44:39.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98407aec0 con 0x7ff98001dc30 2021-03-09T09:44:39.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.032+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:39.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:39.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:39.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:44.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:44.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:44.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099820 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7ff9840469b0 tx=0x7ff98402d3c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:44.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099820 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7ff9840469b0 tx=0x7ff98402d3c0).stop 2021-03-09T09:44:44.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:44.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099820 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:44.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:44.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:44.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:44.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800092b0 con 0x7ff99c0adc20 2021-03-09T09:44:44.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033db0 con 0x7ff99c0613a0 2021-03-09T09:44:44.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:44.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800049c0 con 0x7ff980087a10 2021-03-09T09:44:44.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:44.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:44.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:44.030 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:44.028+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2281 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:44.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:44.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:44.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:44.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:44.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:44.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005390 con 0x7ff980087a10 2021-03-09T09:44:44.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050ae0 con 0x7ff980087a10 2021-03-09T09:44:44.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.032+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7ff9840585c0 tx=0x7ff9840414a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:44.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980087a10 2021-03-09T09:44:44.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984028c10 con 0x7ff980087a10 2021-03-09T09:44:44.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98407aec0 con 0x7ff980087a10 2021-03-09T09:44:44.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.036+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:44.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:44.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:44.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:49.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:49.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:49.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7ff9840585c0 tx=0x7ff9840414a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:49.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7ff9840585c0 tx=0x7ff9840414a0).stop 2021-03-09T09:44:49.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980005520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:49.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980005520 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:49.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:49.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:49.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:49.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800951c0 con 0x7ff99c0adc20 2021-03-09T09:44:49.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037f50 con 0x7ff99c0613a0 2021-03-09T09:44:49.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff98001dc30 2021-03-09T09:44:49.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:49.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:49.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:49.032 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:49.028+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2318 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:49.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:49.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:49.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.032+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:49.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.032+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:49.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.032+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:49.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.032+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:49.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.032+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980036f40 con 0x7ff98001dc30 2021-03-09T09:44:49.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.032+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093d50 con 0x7ff98001dc30 2021-03-09T09:44:49.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.032+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800999c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff99403a870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:49.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff98001dc30 2021-03-09T09:44:49.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069f20 con 0x7ff98001dc30 2021-03-09T09:44:49.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99407a4b0 con 0x7ff98001dc30 2021-03-09T09:44:49.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.036+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:49.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:49.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:49.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:54.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:54.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:54.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800999c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff99403a870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:54.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800999c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff99403a870).stop 2021-03-09T09:44:54.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800999c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:54.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800999c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:54.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:54.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:54.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980011210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:54.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008000 con 0x7ff99c0adc20 2021-03-09T09:44:54.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800092b0 con 0x7ff99c0613a0 2021-03-09T09:44:54.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980040fa0 con 0x7ff980087a10 2021-03-09T09:44:54.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:54.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:54.034 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:54.027+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2357 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:54.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:54.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:54.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.027+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980011210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:54.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.031+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:54.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.031+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:54.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.031+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:54.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.031+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:54.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.031+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d9c0 con 0x7ff980087a10 2021-03-09T09:44:54.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.031+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980036f40 con 0x7ff980087a10 2021-03-09T09:44:54.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.031+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980011210 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7ff98004e920 tx=0x7ff994070150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:54.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.035+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980087a10 2021-03-09T09:44:54.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.035+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405cba0 con 0x7ff980087a10 2021-03-09T09:44:54.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940740e0 con 0x7ff980087a10 2021-03-09T09:44:54.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.039+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:54.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.039+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:54.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:54.039+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:44:55.142 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:44:55.146 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:44:59.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980011210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:59.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980011210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:59.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980011210 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7ff98004e920 tx=0x7ff994070150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:59.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980011210 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7ff98004e920 tx=0x7ff994070150).stop 2021-03-09T09:44:59.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980011210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:59.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980011210 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:59.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:59.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:59.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:59.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052df0 con 0x7ff99c0adc20 2021-03-09T09:44:59.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037f50 con 0x7ff99c0613a0 2021-03-09T09:44:59.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800094b0 con 0x7ff98001dc30 2021-03-09T09:44:59.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:44:59.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:59.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:44:59.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:59.036 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:44:59.030+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2391 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:44:59.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.030+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:59.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.034+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:59.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.034+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:59.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.034+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:59.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.034+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:59.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.034+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003be90 con 0x7ff98001dc30 2021-03-09T09:44:59.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.034+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005390 con 0x7ff98001dc30 2021-03-09T09:44:59.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.034+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001e7d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7ff984000dd0 tx=0x7ff984031b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:59.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.034+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff98001dc30 2021-03-09T09:44:59.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.042+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840a6080 con 0x7ff98001dc30 2021-03-09T09:44:59.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.042+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840587e0 con 0x7ff98001dc30 2021-03-09T09:44:59.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.042+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:59.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.042+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:59.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:44:59.042+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:04.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98001e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:45:04.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98001e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:04.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001e7d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7ff984000dd0 tx=0x7ff984031b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:04.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001e7d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7ff984000dd0 tx=0x7ff984031b80).stop 2021-03-09T09:45:04.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98001e7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:04.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98001e7d0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:04.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:04.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:04.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:04.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071880 con 0x7ff99c0adc20 2021-03-09T09:45:04.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800397c0 con 0x7ff99c0613a0 2021-03-09T09:45:04.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055a60 con 0x7ff980087a10 2021-03-09T09:45:04.037 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:04.030+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2425 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:04.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:04.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:04.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:04.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:04.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.030+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:04.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.034+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:04.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.034+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:04.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.034+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:04.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.034+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:04.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.034+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980008a40 con 0x7ff980087a10 2021-03-09T09:45:04.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.034+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003be90 con 0x7ff980087a10 2021-03-09T09:45:04.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.034+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001ecc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7ff984084ac0 tx=0x7ff98400f1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:04.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.038+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980087a10 2021-03-09T09:45:04.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.038+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840a6020 con 0x7ff980087a10 2021-03-09T09:45:04.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.042+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980087a10 2021-03-09T09:45:04.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.042+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:04.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.042+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:04.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:04.042+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:09.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98001ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:45:09.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98001ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:09.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001ecc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7ff984084ac0 tx=0x7ff98400f1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:09.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001ecc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7ff984084ac0 tx=0x7ff98400f1f0).stop 2021-03-09T09:45:09.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98001ecc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:09.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001ecc0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:09.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:09.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980005de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:09.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:09.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050240 con 0x7ff99c0adc20 2021-03-09T09:45:09.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800049f0 con 0x7ff99c0613a0 2021-03-09T09:45:09.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d210 con 0x7ff98001dc30 2021-03-09T09:45:09.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:09.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:09.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:09.039 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:09.033+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2465 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:09.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:09.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980005de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:09.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:09.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:09.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:09.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:09.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093ef0 con 0x7ff98001dc30 2021-03-09T09:45:09.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980039900 con 0x7ff98001dc30 2021-03-09T09:45:09.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980005de0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7ff98000e460 tx=0x7ff98c003f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:09.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff98001dc30 2021-03-09T09:45:09.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d020 con 0x7ff98001dc30 2021-03-09T09:45:09.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.045+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98001dc30 2021-03-09T09:45:09.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.045+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:09.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.045+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:09.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:09.045+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:14.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980005de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:45:14.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980005de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:14.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980005de0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7ff98000e460 tx=0x7ff98c003f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:14.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980005de0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7ff98000e460 tx=0x7ff98c003f00).stop 2021-03-09T09:45:14.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980005de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:14.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980005de0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:14.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:14.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:14.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:14.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008900 con 0x7ff99c0adc20 2021-03-09T09:45:14.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cb80 con 0x7ff99c0613a0 2021-03-09T09:45:14.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800397c0 con 0x7ff980087a10 2021-03-09T09:45:14.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:14.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:14.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:14.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.033+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:14.041 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:14.033+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2501 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:14.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.037+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:14.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.037+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:14.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.037+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:14.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.037+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:14.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.037+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:14.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.037+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800953c0 con 0x7ff980087a10 2021-03-09T09:45:14.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.037+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980008a40 con 0x7ff980087a10 2021-03-09T09:45:14.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.041+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800996d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff98404e8f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:14.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.041+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98404ec90 con 0x7ff980087a10 2021-03-09T09:45:14.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.041+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98404ec90 con 0x7ff980087a10 2021-03-09T09:45:14.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.045+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984084570 con 0x7ff980087a10 2021-03-09T09:45:14.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.045+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:14.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.045+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:14.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:14.045+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:19.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:19.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:19.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800996d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff98404e8f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:19.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800996d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff98404e8f0).stop 2021-03-09T09:45:19.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800996d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:19.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800996d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:19.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:19.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:19.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:19.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d630 con 0x7ff99c0adc20 2021-03-09T09:45:19.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055cb0 con 0x7ff99c0613a0 2021-03-09T09:45:19.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019d40 con 0x7ff98001dc30 2021-03-09T09:45:19.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:19.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:19.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:19.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:19.042 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:19.036+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2535 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:19.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:19.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:19.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:19.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:19.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:19.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980040390 con 0x7ff98001dc30 2021-03-09T09:45:19.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800953c0 con 0x7ff98001dc30 2021-03-09T09:45:19.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.036+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003350 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7ff984009150 tx=0x7ff984084b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:19.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff98001dc30 2021-03-09T09:45:19.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840319d0 con 0x7ff98001dc30 2021-03-09T09:45:19.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984027110 con 0x7ff98001dc30 2021-03-09T09:45:19.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.044+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:19.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.044+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:19.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:19.044+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:24.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980003350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:24.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980003350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:24.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003350 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7ff984009150 tx=0x7ff984084b60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:24.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003350 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7ff984009150 tx=0x7ff984084b60).stop 2021-03-09T09:45:24.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980003350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:24.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003350 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:24.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:24.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800932d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:24.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:24.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055a60 con 0x7ff99c0adc20 2021-03-09T09:45:24.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013f70 con 0x7ff99c0613a0 2021-03-09T09:45:24.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:24.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cb80 con 0x7ff980087a10 2021-03-09T09:45:24.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:24.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:24.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:24.044 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:24.036+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2571 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:24.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.036+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800932d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:24.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.040+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:24.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.040+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:24.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.040+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:24.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.040+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:24.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.040+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000f310 con 0x7ff980087a10 2021-03-09T09:45:24.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.040+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093ef0 con 0x7ff980087a10 2021-03-09T09:45:24.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.040+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800932d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7ff98c050580 tx=0x7ff98c020ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:24.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff980087a10 2021-03-09T09:45:24.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d080 con 0x7ff980087a10 2021-03-09T09:45:24.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980087a10 2021-03-09T09:45:24.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.048+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:24.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.048+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:24.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:24.048+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:25.205 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:45:25.209 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:45:29.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800932d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:45:29.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800932d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:29.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800932d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7ff98c050580 tx=0x7ff98c020ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:29.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800932d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7ff98c050580 tx=0x7ff98c020ee0).stop 2021-03-09T09:45:29.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800932d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:29.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800932d0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:29.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:29.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:29.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:29.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980040f60 con 0x7ff99c0adc20 2021-03-09T09:45:29.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d630 con 0x7ff99c0613a0 2021-03-09T09:45:29.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010230 con 0x7ff98001dc30 2021-03-09T09:45:29.044 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:29.036+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2608 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:29.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:29.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:29.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:29.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.036+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:29.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.040+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:29.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:29.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:29.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:29.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:29.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800537e0 con 0x7ff98001dc30 2021-03-09T09:45:29.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000f310 con 0x7ff98001dc30 2021-03-09T09:45:29.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.048+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003730 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7ff98c01f790 tx=0x7ff98c079320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:29.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff98001dc30 2021-03-09T09:45:29.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e020 con 0x7ff98001dc30 2021-03-09T09:45:29.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c020b70 con 0x7ff98001dc30 2021-03-09T09:45:29.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.052+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:29.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.056+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:29.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:29.056+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:34.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980003730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:45:34.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980003730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:34.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003730 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7ff98c01f790 tx=0x7ff98c079320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:34.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003730 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7ff98c01f790 tx=0x7ff98c079320).stop 2021-03-09T09:45:34.044 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:34.039+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2641 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:34.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980003730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:34.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980003730 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:34.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:34.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:34.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800044e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:34.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cb80 con 0x7ff99c0adc20 2021-03-09T09:45:34.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980032830 con 0x7ff99c0613a0 2021-03-09T09:45:34.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800300b0 con 0x7ff980087a10 2021-03-09T09:45:34.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:34.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:34.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:34.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:34.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:34.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:34.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:34.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:34.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:34.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800337d0 con 0x7ff980087a10 2021-03-09T09:45:34.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d9c0 con 0x7ff980087a10 2021-03-09T09:45:34.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.039+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800044e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7ff99406a130 tx=0x7ff9940701c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:34.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.043+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980087a10 2021-03-09T09:45:34.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.043+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980087a10 2021-03-09T09:45:34.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.047+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980087a10 2021-03-09T09:45:34.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.047+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:34.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.047+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:34.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:34.047+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:39.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:45:39.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:39.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800044e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7ff99406a130 tx=0x7ff9940701c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:39.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800044e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7ff99406a130 tx=0x7ff9940701c0).stop 2021-03-09T09:45:39.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800044e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:39.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800044e0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:39.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:39.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:39.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:39.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d630 con 0x7ff99c0adc20 2021-03-09T09:45:39.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004fa00 con 0x7ff99c0613a0 2021-03-09T09:45:39.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b880 con 0x7ff98001dc30 2021-03-09T09:45:39.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:39.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:39.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:39.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:39.049 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:39.039+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2677 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:39.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.039+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:39.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.043+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:39.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.043+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:39.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.043+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:39.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.043+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:39.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.043+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000a390 con 0x7ff98001dc30 2021-03-09T09:45:39.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.043+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800537e0 con 0x7ff98001dc30 2021-03-09T09:45:39.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.043+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800996d0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7ff980091490 tx=0x7ff98c08ee70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:39.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.047+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff98001dc30 2021-03-09T09:45:39.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.051+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98001dc30 2021-03-09T09:45:39.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.051+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c084d70 con 0x7ff98001dc30 2021-03-09T09:45:39.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.051+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:39.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.051+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:39.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:39.051+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:44.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:45:44.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:44.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800996d0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7ff980091490 tx=0x7ff98c08ee70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:44.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800996d0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7ff980091490 tx=0x7ff98c08ee70).stop 2021-03-09T09:45:44.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800996d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:44.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800996d0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:44.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:44.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:44.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:44.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010c10 con 0x7ff99c0adc20 2021-03-09T09:45:44.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052ff0 con 0x7ff99c0613a0 2021-03-09T09:45:44.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094b70 con 0x7ff980087a10 2021-03-09T09:45:44.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:44.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:44.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:44.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:44.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.039+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:44.050 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:44.039+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2714 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:44.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.043+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:44.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.043+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:44.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.043+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:44.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.043+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:44.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.043+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002dfc0 con 0x7ff980087a10 2021-03-09T09:45:44.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.043+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980040390 con 0x7ff980087a10 2021-03-09T09:45:44.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.047+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048ea0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984034180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:44.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.051+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019080 con 0x7ff980087a10 2021-03-09T09:45:44.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.051+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984006b70 con 0x7ff980087a10 2021-03-09T09:45:44.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.055+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400f180 con 0x7ff980087a10 2021-03-09T09:45:44.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.055+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:44.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.055+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:44.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:44.055+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:49.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980048ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:49.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980048ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:49.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048ea0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984034180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:49.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048ea0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984034180).stop 2021-03-09T09:45:49.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980048ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:49.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048ea0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:49.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800105a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:49.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:49.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:49.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094b70 con 0x7ff99c0adc20 2021-03-09T09:45:49.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001c040 con 0x7ff99c0613a0 2021-03-09T09:45:49.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980064b80 con 0x7ff98001dc30 2021-03-09T09:45:49.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:49.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:49.050 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:49.043+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2751 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:49.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:49.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:49.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.043+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:49.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.047+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:49.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.047+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:49.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.047+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:49.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.047+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:49.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.047+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094090 con 0x7ff98001dc30 2021-03-09T09:45:49.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.047+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002dfc0 con 0x7ff98001dc30 2021-03-09T09:45:49.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.047+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800105a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98400f2a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:49.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.047+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff98001dc30 2021-03-09T09:45:49.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.051+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984084780 con 0x7ff98001dc30 2021-03-09T09:45:49.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.055+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff98001dc30 2021-03-09T09:45:49.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.055+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:49.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.055+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:49.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:49.055+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:54.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800105a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:45:54.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800105a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:54.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800105a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98400f2a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:54.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800105a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98400f2a0).stop 2021-03-09T09:45:54.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800105a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:54.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800105a0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:54.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:54.050 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:54.043+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2786 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:54.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:54.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:54.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800482e0 con 0x7ff99c0adc20 2021-03-09T09:45:54.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094b70 con 0x7ff99c0613a0 2021-03-09T09:45:54.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019270 con 0x7ff980087a10 2021-03-09T09:45:54.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:54.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:54.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:54.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:54.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.043+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:54.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.047+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:54.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.047+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:54.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.047+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:54.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.047+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:54.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.047+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800295c0 con 0x7ff980087a10 2021-03-09T09:45:54.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.047+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800337d0 con 0x7ff980087a10 2021-03-09T09:45:54.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.047+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004bb10 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7ff98004ec30 tx=0x7ff994009a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:54.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.047+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940671e0 con 0x7ff980087a10 2021-03-09T09:45:54.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.047+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400b9f0 con 0x7ff980087a10 2021-03-09T09:45:54.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.051+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940671e0 con 0x7ff980087a10 2021-03-09T09:45:54.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.051+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:54.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.055+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:54.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:54.055+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:45:55.255 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:45:55.258 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:45:59.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98004bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:59.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98004bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:59.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004bb10 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7ff98004ec30 tx=0x7ff994009a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:59.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004bb10 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7ff98004ec30 tx=0x7ff994009a30).stop 2021-03-09T09:45:59.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98004bb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:59.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004bb10 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:59.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:59.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:59.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:59.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800121e0 con 0x7ff99c0adc20 2021-03-09T09:45:59.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001c040 con 0x7ff99c0613a0 2021-03-09T09:45:59.052 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:45:59.046+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2825 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:45:59.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094b70 con 0x7ff98001dc30 2021-03-09T09:45:59.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:45:59.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:59.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:45:59.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:59.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:59.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:59.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:59.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:59.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:59.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800537e0 con 0x7ff98001dc30 2021-03-09T09:45:59.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094090 con 0x7ff98001dc30 2021-03-09T09:45:59.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7ff984084b30 tx=0x7ff984046cd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:59.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98403ea80 con 0x7ff98001dc30 2021-03-09T09:45:59.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984084780 con 0x7ff98001dc30 2021-03-09T09:45:59.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840270c0 con 0x7ff98001dc30 2021-03-09T09:45:59.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.054+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:59.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.054+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:59.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:45:59.054+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:04.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:04.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:04.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7ff984084b30 tx=0x7ff984046cd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:04.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7ff984084b30 tx=0x7ff984046cd0).stop 2021-03-09T09:46:04.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800993e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:04.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:04.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:04.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:04.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:04.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001f3c0 con 0x7ff99c0adc20 2021-03-09T09:46:04.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800482e0 con 0x7ff99c0613a0 2021-03-09T09:46:04.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094b70 con 0x7ff980087a10 2021-03-09T09:46:04.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:04.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:04.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:04.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:04.056 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:04.046+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2862 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:04.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:04.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:04.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:04.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:04.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:04.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800273a0 con 0x7ff980087a10 2021-03-09T09:46:04.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800537e0 con 0x7ff980087a10 2021-03-09T09:46:04.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7ff9840367a0 tx=0x7ff984076bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:04.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980087a10 2021-03-09T09:46:04.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984084780 con 0x7ff980087a10 2021-03-09T09:46:04.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840471e0 con 0x7ff980087a10 2021-03-09T09:46:04.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.054+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:04.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.058+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:04.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:04.058+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:09.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980048460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:09.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980048460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:09.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7ff9840367a0 tx=0x7ff984076bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:09.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7ff9840367a0 tx=0x7ff984076bc0).stop 2021-03-09T09:46:09.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980048460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:09.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:09.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:09.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:09.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:09.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980021540 con 0x7ff99c0adc20 2021-03-09T09:46:09.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018420 con 0x7ff99c0613a0 2021-03-09T09:46:09.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004d80 con 0x7ff98001dc30 2021-03-09T09:46:09.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:09.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:09.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:09.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:09.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:09.057 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:09.046+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2895 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:09.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:09.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:09.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:09.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:09.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800387e0 con 0x7ff98001dc30 2021-03-09T09:46:09.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.046+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000a390 con 0x7ff98001dc30 2021-03-09T09:46:09.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004bb10 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7ff980038970 tx=0x7ff98c04d230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:09.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff98001dc30 2021-03-09T09:46:09.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e040 con 0x7ff98001dc30 2021-03-09T09:46:09.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.058+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff98001dc30 2021-03-09T09:46:09.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.058+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:09.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.058+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:09.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:09.058+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:14.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:14.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:14.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004bb10 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7ff980038970 tx=0x7ff98c04d230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:14.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004bb10 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7ff980038970 tx=0x7ff98c04d230).stop 2021-03-09T09:46:14.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004bb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:14.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004bb10 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:14.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:14.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:14.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:14.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980040660 con 0x7ff99c0adc20 2021-03-09T09:46:14.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800482e0 con 0x7ff99c0613a0 2021-03-09T09:46:14.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a920 con 0x7ff980087a10 2021-03-09T09:46:14.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:14.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:14.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:14.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:14.059 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:14.050+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2929 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:14.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:14.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:14.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:14.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:14.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:14.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800278d0 con 0x7ff980087a10 2021-03-09T09:46:14.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800387e0 con 0x7ff980087a10 2021-03-09T09:46:14.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001d700 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c016b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:14.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023070 con 0x7ff980087a10 2021-03-09T09:46:14.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e040 con 0x7ff980087a10 2021-03-09T09:46:14.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0162a0 con 0x7ff980087a10 2021-03-09T09:46:14.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.054+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:14.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.058+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:14.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:14.058+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:19.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98001d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:19.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98001d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:19.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001d700 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c016b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:19.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001d700 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c016b20).stop 2021-03-09T09:46:19.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98001d700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:19.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98001d700 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:19.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:19.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:19.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800230e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:19.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:19.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:19.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800947c0 con 0x7ff99c0adc20 2021-03-09T09:46:19.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043620 con 0x7ff99c0613a0 2021-03-09T09:46:19.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980040520 con 0x7ff98001dc30 2021-03-09T09:46:19.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:19.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:19.058 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:19.050+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2967 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:19.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800230e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:19.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:19.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:19.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:19.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:19.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800337d0 con 0x7ff98001dc30 2021-03-09T09:46:19.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800295c0 con 0x7ff98001dc30 2021-03-09T09:46:19.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.050+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800230e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7ff99400b0b0 tx=0x7ff994070320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:19.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.058+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994076800 con 0x7ff98001dc30 2021-03-09T09:46:19.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.058+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940780b0 con 0x7ff98001dc30 2021-03-09T09:46:19.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.058+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99407a4b0 con 0x7ff98001dc30 2021-03-09T09:46:19.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.058+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:19.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.058+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:19.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:19.058+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:24.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800230e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:24.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800230e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:24.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800230e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7ff99400b0b0 tx=0x7ff994070320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:24.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800230e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7ff99400b0b0 tx=0x7ff994070320).stop 2021-03-09T09:46:24.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800230e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:24.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800230e0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:24.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:24.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:24.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:24.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006690 con 0x7ff99c0adc20 2021-03-09T09:46:24.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094b70 con 0x7ff99c0613a0 2021-03-09T09:46:24.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800482e0 con 0x7ff980087a10 2021-03-09T09:46:24.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:24.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:24.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:24.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:24.060 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:24.050+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3002 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:24.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.050+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:24.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:24.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.054+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:24.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:24.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.054+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:24.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800556d0 con 0x7ff980087a10 2021-03-09T09:46:24.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800278d0 con 0x7ff980087a10 2021-03-09T09:46:24.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.054+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800409e0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7ff98c079f00 tx=0x7ff98c017f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:24.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.058+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053020 con 0x7ff980087a10 2021-03-09T09:46:24.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.058+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e080 con 0x7ff980087a10 2021-03-09T09:46:24.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.062+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980087a10 2021-03-09T09:46:24.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.062+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:24.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.062+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:24.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:24.062+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:25.401 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:46:25.404 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:46:29.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:29.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:29.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800409e0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7ff98c079f00 tx=0x7ff98c017f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:29.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800409e0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7ff98c079f00 tx=0x7ff98c017f70).stop 2021-03-09T09:46:29.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800409e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:29.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800409e0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:29.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:29.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:29.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:29.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046fe0 con 0x7ff99c0adc20 2021-03-09T09:46:29.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003ae40 con 0x7ff99c0613a0 2021-03-09T09:46:29.060 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:29.054+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3037 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:29.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003e480 con 0x7ff98001dc30 2021-03-09T09:46:29.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:29.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:29.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:29.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:29.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:29.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:29.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:29.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:29.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:29.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980018780 con 0x7ff98001dc30 2021-03-09T09:46:29.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800273a0 con 0x7ff98001dc30 2021-03-09T09:46:29.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7ff98401f010 tx=0x7ff984058a40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:29.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840270a0 con 0x7ff98001dc30 2021-03-09T09:46:29.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.062+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984006aa0 con 0x7ff98001dc30 2021-03-09T09:46:29.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.062+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840270a0 con 0x7ff98001dc30 2021-03-09T09:46:29.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.062+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:29.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.062+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:29.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:29.062+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:34.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:34.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:34.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7ff98401f010 tx=0x7ff984058a40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:34.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7ff98401f010 tx=0x7ff984058a40).stop 2021-03-09T09:46:34.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800993e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:34.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800993e0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:34.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:34.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:34.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:34.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:34.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:34.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057010 con 0x7ff99c0adc20 2021-03-09T09:46:34.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a890 con 0x7ff99c0613a0 2021-03-09T09:46:34.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003ddc0 con 0x7ff980087a10 2021-03-09T09:46:34.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:34.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:34.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:34.063 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:34.054+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3075 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:34.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:34.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:34.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:34.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:34.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800387e0 con 0x7ff980087a10 2021-03-09T09:46:34.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800556d0 con 0x7ff980087a10 2021-03-09T09:46:34.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.054+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095bc0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7ff98c04aa60 tx=0x7ff98c029c40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:34.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.058+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff980087a10 2021-03-09T09:46:34.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.058+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e040 con 0x7ff980087a10 2021-03-09T09:46:34.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.062+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980087a10 2021-03-09T09:46:34.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.062+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:34.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.062+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:34.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:34.062+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:39.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980095bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:39.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980095bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:39.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095bc0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7ff98c04aa60 tx=0x7ff98c029c40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:39.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095bc0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7ff98c04aa60 tx=0x7ff98c029c40).stop 2021-03-09T09:46:39.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980095bc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:39.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980095bc0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:39.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:39.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:39.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:39.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094b70 con 0x7ff99c0adc20 2021-03-09T09:46:39.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017180 con 0x7ff99c0613a0 2021-03-09T09:46:39.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001aa30 con 0x7ff98001dc30 2021-03-09T09:46:39.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:39.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:39.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:39.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:39.063 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:39.053+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3109 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:39.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:39.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:39.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:39.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:39.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:39.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff98001dc30 2021-03-09T09:46:39.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.053+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980018780 con 0x7ff98001dc30 2021-03-09T09:46:39.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.057+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800980c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff9840423c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:39.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984042210 con 0x7ff98001dc30 2021-03-09T09:46:39.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98403eb50 con 0x7ff98001dc30 2021-03-09T09:46:39.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff98001dc30 2021-03-09T09:46:39.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.061+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:39.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.061+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:39.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:39.061+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:44.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:46:44.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:44.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800980c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff9840423c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:44.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800980c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff9840423c0).stop 2021-03-09T09:46:44.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff9800980c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:44.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff9800980c0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:44.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:44.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:44.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:44.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f3e0 con 0x7ff99c0adc20 2021-03-09T09:46:44.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800226f0 con 0x7ff99c0613a0 2021-03-09T09:46:44.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005730 con 0x7ff980087a10 2021-03-09T09:46:44.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:44.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:44.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:44.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:44.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.053+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:44.065 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:44.053+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3149 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:44.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.057+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:44.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.057+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:44.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.057+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:44.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.057+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:44.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.057+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008aff0 con 0x7ff980087a10 2021-03-09T09:46:44.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.057+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800387e0 con 0x7ff980087a10 2021-03-09T09:46:44.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.057+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a760 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7ff98c029af0 tx=0x7ff98c084f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:44.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980087a10 2021-03-09T09:46:44.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e040 con 0x7ff980087a10 2021-03-09T09:46:44.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c029d10 con 0x7ff980087a10 2021-03-09T09:46:44.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.065+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:44.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.069+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:44.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:44.069+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:49.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98009a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:49.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98009a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:49.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a760 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7ff98c029af0 tx=0x7ff98c084f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:49.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a760 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7ff98c029af0 tx=0x7ff98c084f80).stop 2021-03-09T09:46:49.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98009a760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:49.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a760 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:49.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:49.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:49.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980097bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:49.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d780 con 0x7ff99c0adc20 2021-03-09T09:46:49.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009e30 con 0x7ff99c0613a0 2021-03-09T09:46:49.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d460 con 0x7ff98001dc30 2021-03-09T09:46:49.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:49.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:49.066 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:49.057+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3182 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:49.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:49.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:49.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980097bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:49.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:49.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:49.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:49.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:49.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026c30 con 0x7ff98001dc30 2021-03-09T09:46:49.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800337d0 con 0x7ff98001dc30 2021-03-09T09:46:49.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.061+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980097bd0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff9940701c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:49.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98001dc30 2021-03-09T09:46:49.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98001dc30 2021-03-09T09:46:49.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99407a4b0 con 0x7ff98001dc30 2021-03-09T09:46:49.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.069+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:49.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.069+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:49.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:49.069+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:54.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980097bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:54.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980097bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:54.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980097bd0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff9940701c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:54.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980097bd0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff9940701c0).stop 2021-03-09T09:46:54.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980097bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:54.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980097bd0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:54.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:54.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:54.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:54.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001cc30 con 0x7ff99c0adc20 2021-03-09T09:46:54.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f560 con 0x7ff99c0613a0 2021-03-09T09:46:54.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800482e0 con 0x7ff980087a10 2021-03-09T09:46:54.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:54.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:54.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:54.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:54.069 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:54.057+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3219 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:54.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:54.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:54.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:54.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:54.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:54.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800132e0 con 0x7ff980087a10 2021-03-09T09:46:54.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026c30 con 0x7ff980087a10 2021-03-09T09:46:54.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.061+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a0d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406a070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:54.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980087a10 2021-03-09T09:46:54.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940750c0 con 0x7ff980087a10 2021-03-09T09:46:54.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940aea90 con 0x7ff980087a10 2021-03-09T09:46:54.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.065+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:54.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.065+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:54.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:54.065+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:46:55.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:46:55.453 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:46:59.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98009a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:59.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98009a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:59.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a0d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406a070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:59.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a0d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406a070).stop 2021-03-09T09:46:59.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98009a0d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:59.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009a0d0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:59.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:59.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980096e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:59.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:59.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033db0 con 0x7ff99c0adc20 2021-03-09T09:46:59.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005730 con 0x7ff99c0613a0 2021-03-09T09:46:59.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f560 con 0x7ff98001dc30 2021-03-09T09:46:59.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:46:59.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:59.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:46:59.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:59.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.061+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980096e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:59.070 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:46:59.061+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3254 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:46:59.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.065+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:59.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.065+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:59.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.065+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:59.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.065+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:59.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.065+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800337d0 con 0x7ff98001dc30 2021-03-09T09:46:59.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.065+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008aff0 con 0x7ff98001dc30 2021-03-09T09:46:59.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.065+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980096e50 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7ff98004ce50 tx=0x7ff98c029b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:59.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053020 con 0x7ff98001dc30 2021-03-09T09:46:59.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff98001dc30 2021-03-09T09:46:59.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98001dc30 2021-03-09T09:46:59.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.073+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:59.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.073+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:59.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:46:59.073+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:04.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980096e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:47:04.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980096e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:04.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980096e50 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7ff98004ce50 tx=0x7ff98c029b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:04.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980096e50 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7ff98004ce50 tx=0x7ff98c029b90).stop 2021-03-09T09:47:04.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980096e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:04.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980096e50 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:04.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:04.068 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:04.061+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3289 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:04.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:04.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:04.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050440 con 0x7ff99c0adc20 2021-03-09T09:47:04.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094810 con 0x7ff99c0613a0 2021-03-09T09:47:04.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001f3c0 con 0x7ff980087a10 2021-03-09T09:47:04.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:04.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:04.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:04.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:04.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.061+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:04.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.065+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:04.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.065+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:04.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.065+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:04.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.065+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:04.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.065+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009290 con 0x7ff980087a10 2021-03-09T09:47:04.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.065+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff980087a10 2021-03-09T09:47:04.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.065+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099350 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7ff984078010 tx=0x7ff98403eaf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:04.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019090 con 0x7ff980087a10 2021-03-09T09:47:04.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984006aa0 con 0x7ff980087a10 2021-03-09T09:47:04.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff980087a10 2021-03-09T09:47:04.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.073+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:04.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.073+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:04.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:04.073+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:09.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980099350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:09.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980099350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:09.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099350 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7ff984078010 tx=0x7ff98403eaf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:09.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099350 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7ff984078010 tx=0x7ff98403eaf0).stop 2021-03-09T09:47:09.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980099350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:09.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980099350 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:09.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:09.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:09.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:09.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033db0 con 0x7ff99c0adc20 2021-03-09T09:47:09.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008060 con 0x7ff99c0613a0 2021-03-09T09:47:09.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008f6b0 con 0x7ff98001dc30 2021-03-09T09:47:09.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.061+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:09.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.065+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:09.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.065+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:09.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.065+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:09.072 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:09.065+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3323 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:09.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:09.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:09.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:09.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:09.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:09.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004a910 con 0x7ff98001dc30 2021-03-09T09:47:09.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800132e0 con 0x7ff98001dc30 2021-03-09T09:47:09.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.069+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004d880 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff99403b110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:09.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98001dc30 2021-03-09T09:47:09.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.077+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98001dc30 2021-03-09T09:47:09.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.077+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98001dc30 2021-03-09T09:47:09.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.077+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:09.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.077+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:09.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:09.077+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:14.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:47:14.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:14.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004d880 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff99403b110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:14.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004d880 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff99403b110).stop 2021-03-09T09:47:14.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98004d880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:14.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98004d880 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:14.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:14.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:14.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:14.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:14.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:14.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005730 con 0x7ff99c0adc20 2021-03-09T09:47:14.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037890 con 0x7ff99c0613a0 2021-03-09T09:47:14.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050440 con 0x7ff980087a10 2021-03-09T09:47:14.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:14.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:14.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:14.074 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:14.065+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3364 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:14.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:14.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:14.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:14.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:14.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800435d0 con 0x7ff980087a10 2021-03-09T09:47:14.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004a910 con 0x7ff980087a10 2021-03-09T09:47:14.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.069+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098e60 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7ff994066e60 tx=0x7ff994034c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:14.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940084a0 con 0x7ff980087a10 2021-03-09T09:47:14.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994008600 con 0x7ff980087a10 2021-03-09T09:47:14.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994008810 con 0x7ff980087a10 2021-03-09T09:47:14.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.073+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:14.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.073+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:14.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:14.073+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:19.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980098e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:19.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980098e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:19.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098e60 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7ff994066e60 tx=0x7ff994034c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:19.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098e60 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7ff994066e60 tx=0x7ff994034c30).stop 2021-03-09T09:47:19.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980098e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:19.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980098e60 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:19.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:19.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:19.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:19.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092a40 con 0x7ff99c0adc20 2021-03-09T09:47:19.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033db0 con 0x7ff99c0613a0 2021-03-09T09:47:19.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c760 con 0x7ff98001dc30 2021-03-09T09:47:19.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:19.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:19.075 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:19.065+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3398 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:19.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:19.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.069+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:19.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.069+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:19.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.069+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:19.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.069+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:19.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.069+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:19.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.069+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:19.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.069+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff98001dc30 2021-03-09T09:47:19.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.069+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800337d0 con 0x7ff98001dc30 2021-03-09T09:47:19.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.069+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009b6a0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c022f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:19.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053020 con 0x7ff98001dc30 2021-03-09T09:47:19.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98001dc30 2021-03-09T09:47:19.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.077+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff98001dc30 2021-03-09T09:47:19.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.077+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:19.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.077+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:19.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:19.077+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:24.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98009b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:24.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98009b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:24.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009b6a0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c022f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:24.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009b6a0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c022f60).stop 2021-03-09T09:47:24.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98009b6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:24.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009b6a0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:24.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:24.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:24.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:24.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980028d40 con 0x7ff99c0adc20 2021-03-09T09:47:24.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000e700 con 0x7ff99c0613a0 2021-03-09T09:47:24.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088b20 con 0x7ff980087a10 2021-03-09T09:47:24.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:24.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:24.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:24.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:24.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:24.077 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:24.069+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3435 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:24.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:24.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:24.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:24.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:24.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020b20 con 0x7ff980087a10 2021-03-09T09:47:24.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.069+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009290 con 0x7ff980087a10 2021-03-09T09:47:24.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.073+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097c50 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7ff984047be0 tx=0x7ff98400f180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:24.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980087a10 2021-03-09T09:47:24.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984047a50 con 0x7ff980087a10 2021-03-09T09:47:24.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.077+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff980087a10 2021-03-09T09:47:24.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.077+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:24.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.077+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:24.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:24.077+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:25.512 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:47:25.515 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:47:29.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980097c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:29.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980097c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:29.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097c50 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7ff984047be0 tx=0x7ff98400f180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:29.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097c50 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7ff984047be0 tx=0x7ff98400f180).stop 2021-03-09T09:47:29.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980097c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:29.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980097c50 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:29.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:29.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:29.076 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:29.069+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3468 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:29.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:29.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055fd0 con 0x7ff99c0adc20 2021-03-09T09:47:29.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050440 con 0x7ff99c0613a0 2021-03-09T09:47:29.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043d60 con 0x7ff98001dc30 2021-03-09T09:47:29.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:29.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:29.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:29.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:29.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.069+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:29.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.073+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:29.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.073+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:29.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.073+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:29.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.073+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:29.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.073+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002b580 con 0x7ff98001dc30 2021-03-09T09:47:29.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.073+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020b20 con 0x7ff98001dc30 2021-03-09T09:47:29.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.073+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009a150 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7ff9840035a0 tx=0x7ff9840a5c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:29.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.077+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019020 con 0x7ff98001dc30 2021-03-09T09:47:29.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.077+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984047840 con 0x7ff98001dc30 2021-03-09T09:47:29.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.081+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840447a0 con 0x7ff98001dc30 2021-03-09T09:47:29.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.081+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:29.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.081+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:29.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:29.081+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:34.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98009a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:34.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98009a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:34.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009a150 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7ff9840035a0 tx=0x7ff9840a5c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:34.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009a150 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7ff9840035a0 tx=0x7ff9840a5c20).stop 2021-03-09T09:47:34.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98009a150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:34.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009a150 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:34.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:34.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:34.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:34.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003c020 con 0x7ff99c0adc20 2021-03-09T09:47:34.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027350 con 0x7ff99c0613a0 2021-03-09T09:47:34.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092200 con 0x7ff980087a10 2021-03-09T09:47:34.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:34.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.069+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:34.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.073+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:34.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.073+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:34.079 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:34.073+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3506 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:34.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.073+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:34.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.073+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:34.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.073+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:34.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.073+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:34.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.073+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:34.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.073+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800090d0 con 0x7ff980087a10 2021-03-09T09:47:34.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.073+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002b580 con 0x7ff980087a10 2021-03-09T09:47:34.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.073+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003d960 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7ff9840a5f50 tx=0x7ff984047c10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:34.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.077+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019080 con 0x7ff980087a10 2021-03-09T09:47:34.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.077+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984036070 con 0x7ff980087a10 2021-03-09T09:47:34.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.081+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984036070 con 0x7ff980087a10 2021-03-09T09:47:34.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.081+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:34.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.081+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:34.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:34.081+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:39.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98003d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:47:39.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98003d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:39.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003d960 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7ff9840a5f50 tx=0x7ff984047c10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:39.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003d960 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7ff9840a5f50 tx=0x7ff984047c10).stop 2021-03-09T09:47:39.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98003d960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:39.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98003d960 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:39.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:39.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:39.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:39.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800213f0 con 0x7ff99c0adc20 2021-03-09T09:47:39.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050440 con 0x7ff99c0613a0 2021-03-09T09:47:39.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093c80 con 0x7ff98001dc30 2021-03-09T09:47:39.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:39.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:39.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:39.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:39.081 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:39.072+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3542 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:39.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.072+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:39.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:39.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.076+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:39.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:39.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.076+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:39.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980021530 con 0x7ff98001dc30 2021-03-09T09:47:39.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800090d0 con 0x7ff98001dc30 2021-03-09T09:47:39.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.076+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009c650 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7ff9840290c0 tx=0x7ff98400f260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:39.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840860e0 con 0x7ff98001dc30 2021-03-09T09:47:39.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98402e9a0 con 0x7ff98001dc30 2021-03-09T09:47:39.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840447a0 con 0x7ff98001dc30 2021-03-09T09:47:39.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.084+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:39.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.084+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:39.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:39.084+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:44.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98009c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:44.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98009c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:44.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009c650 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7ff9840290c0 tx=0x7ff98400f260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:44.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009c650 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7ff9840290c0 tx=0x7ff98400f260).stop 2021-03-09T09:47:44.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff98009c650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:44.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff98009c650 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:44.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:44.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:44.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800230c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:44.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006ca0 con 0x7ff99c0adc20 2021-03-09T09:47:44.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d2a0 con 0x7ff99c0613a0 2021-03-09T09:47:44.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d720 con 0x7ff980087a10 2021-03-09T09:47:44.081 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:44.072+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3582 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:44.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:44.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:44.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:44.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:44.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.072+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800230c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:44.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:44.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:44.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:44.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:44.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004a740 con 0x7ff980087a10 2021-03-09T09:47:44.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800435d0 con 0x7ff980087a10 2021-03-09T09:47:44.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.080+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800230c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994035040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:44.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980087a10 2021-03-09T09:47:44.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994071ee0 con 0x7ff980087a10 2021-03-09T09:47:44.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980087a10 2021-03-09T09:47:44.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.084+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:44.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.084+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:44.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:44.084+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:49.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800230c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:49.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800230c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:49.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800230c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994035040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:49.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800230c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994035040).stop 2021-03-09T09:47:49.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff9800230c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:49.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff9800230c0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:49.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:49.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:49.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:49.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033db0 con 0x7ff99c0adc20 2021-03-09T09:47:49.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002fa10 con 0x7ff99c0613a0 2021-03-09T09:47:49.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050440 con 0x7ff98001dc30 2021-03-09T09:47:49.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:49.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.072+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:49.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:49.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.076+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:49.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:49.084 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:49.076+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3613 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:49.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:49.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:49.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:49.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:49.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008bc60 con 0x7ff98001dc30 2021-03-09T09:47:49.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004a740 con 0x7ff98001dc30 2021-03-09T09:47:49.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c60 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7ff9800966e0 tx=0x7ff994071a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:49.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff98001dc30 2021-03-09T09:47:49.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99403e0d0 con 0x7ff98001dc30 2021-03-09T09:47:49.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99403e0d0 con 0x7ff98001dc30 2021-03-09T09:47:49.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.084+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:49.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.088+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:49.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:49.088+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:54.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:47:54.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:54.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c60 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7ff9800966e0 tx=0x7ff994071a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:54.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c60 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7ff9800966e0 tx=0x7ff994071a70).stop 2021-03-09T09:47:54.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:54.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c60 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:54.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:54.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:54.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:54.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001f3c0 con 0x7ff99c0adc20 2021-03-09T09:47:54.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051eb0 con 0x7ff99c0613a0 2021-03-09T09:47:54.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050440 con 0x7ff980087a10 2021-03-09T09:47:54.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:54.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:54.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:54.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:54.085 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:54.076+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3652 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:54.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:54.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:54.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:54.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:54.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:54.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004a910 con 0x7ff980087a10 2021-03-09T09:47:54.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980021530 con 0x7ff980087a10 2021-03-09T09:47:54.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.080+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004f460 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7ff9840044a0 tx=0x7ff98405ac60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:54.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019090 con 0x7ff980087a10 2021-03-09T09:47:54.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840587e0 con 0x7ff980087a10 2021-03-09T09:47:54.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984037000 con 0x7ff980087a10 2021-03-09T09:47:54.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.088+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:54.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.088+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:54.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:54.088+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:47:55.554 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:47:55.578 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:47:59.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98004f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:59.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98004f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:59.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004f460 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7ff9840044a0 tx=0x7ff98405ac60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:59.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004f460 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7ff9840044a0 tx=0x7ff98405ac60).stop 2021-03-09T09:47:59.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98004f460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:59.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98004f460 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:59.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:59.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:59.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:59.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016a70 con 0x7ff99c0adc20 2021-03-09T09:47:59.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003890 con 0x7ff99c0613a0 2021-03-09T09:47:59.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980042320 con 0x7ff98001dc30 2021-03-09T09:47:59.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:47:59.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:47:59.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:59.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:59.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.076+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:59.087 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:47:59.076+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3684 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:47:59.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.080+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:59.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.080+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:59.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.080+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:59.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.080+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:59.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.080+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004a780 con 0x7ff98001dc30 2021-03-09T09:47:59.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.080+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004a910 con 0x7ff98001dc30 2021-03-09T09:47:59.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.080+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984047cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:59.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff98001dc30 2021-03-09T09:47:59.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff98001dc30 2021-03-09T09:47:59.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840a5d70 con 0x7ff98001dc30 2021-03-09T09:47:59.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.088+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:59.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.088+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:59.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:47:59.088+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:04.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:48:04.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:04.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984047cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:04.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984047cf0).stop 2021-03-09T09:48:04.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:04.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:04.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:04.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:04.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:04.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000e310 con 0x7ff99c0adc20 2021-03-09T09:48:04.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800907d0 con 0x7ff99c0613a0 2021-03-09T09:48:04.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033330 con 0x7ff980087a10 2021-03-09T09:48:04.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:04.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:04.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:04.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:04.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:04.088 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:04.080+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3723 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:04.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:04.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:04.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:04.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:04.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001fa90 con 0x7ff980087a10 2021-03-09T09:48:04.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.080+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008bc60 con 0x7ff980087a10 2021-03-09T09:48:04.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.084+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7ff994034d70 tx=0x7ff994081670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:04.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980087a10 2021-03-09T09:48:04.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980087a10 2021-03-09T09:48:04.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980087a10 2021-03-09T09:48:04.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.088+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:04.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.088+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:04.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:04.088+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:09.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980048460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:09.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980048460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:09.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7ff994034d70 tx=0x7ff994081670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:09.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7ff994034d70 tx=0x7ff994081670).stop 2021-03-09T09:48:09.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff980048460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:09.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff980048460 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:09.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:09.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:09.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:09.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033db0 con 0x7ff99c0adc20 2021-03-09T09:48:09.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036d80 con 0x7ff99c0613a0 2021-03-09T09:48:09.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002f4d0 con 0x7ff98001dc30 2021-03-09T09:48:09.088 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:09.080+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3755 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:09.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:09.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:09.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:09.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:09.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.080+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:09.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:09.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:09.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:09.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:09.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004a910 con 0x7ff98001dc30 2021-03-09T09:48:09.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff98001dc30 2021-03-09T09:48:09.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7ff98c00c010 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:09.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053030 con 0x7ff98001dc30 2021-03-09T09:48:09.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e080 con 0x7ff98001dc30 2021-03-09T09:48:09.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff98001dc30 2021-03-09T09:48:09.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.092+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:09.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.092+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:09.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:09.092+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:14.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:14.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:14.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7ff98c00c010 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:14.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7ff98c00c010 tx=0x7ff98c003e80).stop 2021-03-09T09:48:14.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:14.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:14.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:14.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:14.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:14.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094280 con 0x7ff99c0adc20 2021-03-09T09:48:14.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056da0 con 0x7ff99c0613a0 2021-03-09T09:48:14.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095930 con 0x7ff980087a10 2021-03-09T09:48:14.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:14.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:14.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:14.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:14.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.080+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:14.091 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:14.080+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3796 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:14.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:14.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:14.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:14.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:14.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004f210 con 0x7ff980087a10 2021-03-09T09:48:14.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004a910 con 0x7ff980087a10 2021-03-09T09:48:14.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009c140 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7ff98c08e890 tx=0x7ff98c0791c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:14.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff980087a10 2021-03-09T09:48:14.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e030 con 0x7ff980087a10 2021-03-09T09:48:14.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c099d90 con 0x7ff980087a10 2021-03-09T09:48:14.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.092+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:14.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.092+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:14.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:14.092+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:19.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98009c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:19.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98009c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:19.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009c140 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7ff98c08e890 tx=0x7ff98c0791c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:19.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009c140 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7ff98c08e890 tx=0x7ff98c0791c0).stop 2021-03-09T09:48:19.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 msgr2=0x7ff98009c140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:19.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980087a10 0x7ff98009c140 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:19.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:19.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:19.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:19.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008af0 con 0x7ff99c0adc20 2021-03-09T09:48:19.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980040f60 con 0x7ff99c0613a0 2021-03-09T09:48:19.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095930 con 0x7ff98001dc30 2021-03-09T09:48:19.091 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:19.084+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3827 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:19.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:19.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:19.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:19.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:19.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.084+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:19.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.088+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:19.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.088+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:19.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.088+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:19.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.088+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:19.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.088+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004e600 con 0x7ff98001dc30 2021-03-09T09:48:19.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.088+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004a780 con 0x7ff98001dc30 2021-03-09T09:48:19.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.088+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7ff98400f230 tx=0x7ff984058340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:19.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984058890 con 0x7ff98001dc30 2021-03-09T09:48:19.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984058890 con 0x7ff98001dc30 2021-03-09T09:48:19.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.096+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840a5d70 con 0x7ff98001dc30 2021-03-09T09:48:19.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.096+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:19.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.096+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:19.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:19.096+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:24.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:24.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:24.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7ff98400f230 tx=0x7ff984058340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:24.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7ff98400f230 tx=0x7ff984058340).stop 2021-03-09T09:48:24.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:24.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:24.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:24.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98003c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:24.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:24.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800392e0 con 0x7ff99c0adc20 2021-03-09T09:48:24.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800199f0 con 0x7ff99c0613a0 2021-03-09T09:48:24.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800496e0 con 0x7ff98003bce0 2021-03-09T09:48:24.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:24.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:24.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:24.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:24.094 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:24.084+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3864 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:24.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98003c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:24.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:24.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:24.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:24.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:24.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004dda0 con 0x7ff98003bce0 2021-03-09T09:48:24.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004f210 con 0x7ff98003bce0 2021-03-09T09:48:24.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98003c0c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7ff98c023040 tx=0x7ff98c0797b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:24.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff98003bce0 2021-03-09T09:48:24.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98003bce0 2021-03-09T09:48:24.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98003bce0 2021-03-09T09:48:24.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.092+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:24.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.096+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:24.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:24.096+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:25.642 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:48:25.646 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:48:29.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98003c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:29.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98003c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:29.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98003c0c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7ff98c023040 tx=0x7ff98c0797b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:29.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98003c0c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7ff98c023040 tx=0x7ff98c0797b0).stop 2021-03-09T09:48:29.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98003c0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:29.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98003c0c0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:29.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:29.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:29.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:29.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004c5d0 con 0x7ff99c0adc20 2021-03-09T09:48:29.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ea30 con 0x7ff99c0613a0 2021-03-09T09:48:29.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008af0 con 0x7ff98001dc30 2021-03-09T09:48:29.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:29.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:29.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.084+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:29.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.088+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:29.095 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:29.088+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3900 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:29.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:29.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:29.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:29.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:29.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:29.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800886f0 con 0x7ff98001dc30 2021-03-09T09:48:29.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004dda0 con 0x7ff98001dc30 2021-03-09T09:48:29.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0x7ff98c016c80 tx=0x7ff98c0336e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:29.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053020 con 0x7ff98001dc30 2021-03-09T09:48:29.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98001dc30 2021-03-09T09:48:29.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.096+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98001dc30 2021-03-09T09:48:29.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.096+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:29.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.096+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:29.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:29.096+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:34.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:48:34.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:34.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0x7ff98c016c80 tx=0x7ff98c0336e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:34.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0x7ff98c016c80 tx=0x7ff98c0336e0).stop 2021-03-09T09:48:34.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:34.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:34.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:34.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:34.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:34.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002da90 con 0x7ff99c0adc20 2021-03-09T09:48:34.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094140 con 0x7ff99c0613a0 2021-03-09T09:48:34.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a960 con 0x7ff98003bce0 2021-03-09T09:48:34.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:34.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:34.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:34.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:34.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:34.097 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:34.088+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3939 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:34.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:34.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:34.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:34.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:34.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008eb70 con 0x7ff98003bce0 2021-03-09T09:48:34.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001fa90 con 0x7ff98003bce0 2021-03-09T09:48:34.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.092+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048460 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7ff980047150 tx=0x7ff994069ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:34.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff98003bce0 2021-03-09T09:48:34.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98003bce0 2021-03-09T09:48:34.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.096+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98003bce0 2021-03-09T09:48:34.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.096+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:34.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.096+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:34.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:34.096+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:39.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980048460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:39.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980048460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:39.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048460 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7ff980047150 tx=0x7ff994069ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:39.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048460 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7ff980047150 tx=0x7ff994069ca0).stop 2021-03-09T09:48:39.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980048460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:39.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048460 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:39.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:39.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:39.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:39.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095930 con 0x7ff99c0adc20 2021-03-09T09:48:39.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002f810 con 0x7ff99c0613a0 2021-03-09T09:48:39.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001cac0 con 0x7ff98001dc30 2021-03-09T09:48:39.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:39.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:39.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:39.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:39.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:39.100 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:39.088+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3978 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:39.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.092+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:39.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.092+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:39.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.092+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:39.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.092+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:39.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.092+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff98001dc30 2021-03-09T09:48:39.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.092+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008eb70 con 0x7ff98001dc30 2021-03-09T09:48:39.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.092+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7ff9800600c0 tx=0x7ff99403ba90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:39.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.096+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff98001dc30 2021-03-09T09:48:39.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.096+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940780d0 con 0x7ff98001dc30 2021-03-09T09:48:39.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994008d30 con 0x7ff98001dc30 2021-03-09T09:48:39.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.100+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:39.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.100+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:39.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:39.100+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:44.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:48:44.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:44.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7ff9800600c0 tx=0x7ff99403ba90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:44.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7ff9800600c0 tx=0x7ff99403ba90).stop 2021-03-09T09:48:44.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:44.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c40 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:44.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:44.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:44.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:44.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800038d0 con 0x7ff99c0adc20 2021-03-09T09:48:44.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037330 con 0x7ff99c0613a0 2021-03-09T09:48:44.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005750 con 0x7ff98003bce0 2021-03-09T09:48:44.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:44.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:44.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:44.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:44.100 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:44.091+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4012 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:44.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:44.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:44.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:44.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:44.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:44.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001e990 con 0x7ff98003bce0 2021-03-09T09:48:44.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.091+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800886f0 con 0x7ff98003bce0 2021-03-09T09:48:44.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800487e0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7ff98c0a02d0 tx=0x7ff98c033670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:44.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff98003bce0 2021-03-09T09:48:44.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d050 con 0x7ff98003bce0 2021-03-09T09:48:44.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98003bce0 2021-03-09T09:48:44.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.103+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:44.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.103+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:44.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:44.103+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:49.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:49.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:49.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800487e0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7ff98c0a02d0 tx=0x7ff98c033670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:49.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800487e0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7ff98c0a02d0 tx=0x7ff98c033670).stop 2021-03-09T09:48:49.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800487e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:49.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800487e0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:49.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:49.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:49.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:49.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a960 con 0x7ff99c0adc20 2021-03-09T09:48:49.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002df30 con 0x7ff99c0613a0 2021-03-09T09:48:49.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095930 con 0x7ff98001dc30 2021-03-09T09:48:49.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:49.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:49.102 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:49.091+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4045 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:49.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:49.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:49.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.091+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:49.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:49.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:49.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:49.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:49.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000cb60 con 0x7ff98001dc30 2021-03-09T09:48:49.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001e990 con 0x7ff98001dc30 2021-03-09T09:48:49.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7ff98c033460 tx=0x7ff98c022130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:49.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff98001dc30 2021-03-09T09:48:49.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04d040 con 0x7ff98001dc30 2021-03-09T09:48:49.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98001dc30 2021-03-09T09:48:49.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.099+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:49.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.103+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:49.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:49.103+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:54.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:54.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:54.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7ff98c033460 tx=0x7ff98c022130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:54.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7ff98c033460 tx=0x7ff98c022130).stop 2021-03-09T09:48:54.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:54.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:54.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:54.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:54.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:54.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044d70 con 0x7ff99c0adc20 2021-03-09T09:48:54.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097530 con 0x7ff99c0613a0 2021-03-09T09:48:54.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800319c0 con 0x7ff98003bce0 2021-03-09T09:48:54.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:54.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:54.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:54.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:54.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:54.105 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:54.095+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4080 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:54.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:54.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:54.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:54.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:54.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800226a0 con 0x7ff98003bce0 2021-03-09T09:48:54.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff98003bce0 2021-03-09T09:48:54.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048c90 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994069d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:54.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99403b0e0 con 0x7ff98003bce0 2021-03-09T09:48:54.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98003bce0 2021-03-09T09:48:54.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff98003bce0 2021-03-09T09:48:54.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.099+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:54.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.103+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:54.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:54.103+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:48:55.797 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:48:55.800 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:48:59.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980048c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:59.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980048c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:59.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048c90 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994069d10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:59.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048c90 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994069d10).stop 2021-03-09T09:48:59.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980048c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:59.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980048c90 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:59.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:59.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:59.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:59.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002f810 con 0x7ff99c0adc20 2021-03-09T09:48:59.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980096340 con 0x7ff99c0613a0 2021-03-09T09:48:59.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095930 con 0x7ff98001dc30 2021-03-09T09:48:59.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:48:59.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:59.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:59.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:48:59.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:59.105 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:48:59.095+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4115 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:48:59.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:59.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:59.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:59.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:59.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054c40 con 0x7ff98001dc30 2021-03-09T09:48:59.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000cb60 con 0x7ff98001dc30 2021-03-09T09:48:59.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.099+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7ff98c0066d0 tx=0x7ff98c003f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:59.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053050 con 0x7ff98001dc30 2021-03-09T09:48:59.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98001dc30 2021-03-09T09:48:59.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98001dc30 2021-03-09T09:48:59.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.103+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:59.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.107+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:59.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:48:59.107+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:04.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:04.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:04.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7ff98c0066d0 tx=0x7ff98c003f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:04.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7ff98c0066d0 tx=0x7ff98c003f30).stop 2021-03-09T09:49:04.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:04.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099c00 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:04.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98000bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:04.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:04.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:04.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003b920 con 0x7ff99c0adc20 2021-03-09T09:49:04.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980021c50 con 0x7ff99c0613a0 2021-03-09T09:49:04.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a960 con 0x7ff98003bce0 2021-03-09T09:49:04.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:04.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:04.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:04.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:04.106 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:04.095+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4156 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:04.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.095+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98000bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:04.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.099+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:04.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.099+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:04.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.099+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:04.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.099+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:04.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.099+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980042210 con 0x7ff98003bce0 2021-03-09T09:49:04.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.099+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004e600 con 0x7ff98003bce0 2021-03-09T09:49:04.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.099+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98000bfb0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840295b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:04.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984027280 con 0x7ff98003bce0 2021-03-09T09:49:04.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840039d0 con 0x7ff98003bce0 2021-03-09T09:49:04.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984003b40 con 0x7ff98003bce0 2021-03-09T09:49:04.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.107+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:04.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.107+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:04.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:04.107+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:09.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98000bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:09.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98000bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:09.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98000bfb0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840295b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:09.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98000bfb0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840295b0).stop 2021-03-09T09:49:09.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98000bfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:09.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98000bfb0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:09.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:09.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:09.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:09.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:09.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:09.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ec30 con 0x7ff99c0adc20 2021-03-09T09:49:09.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dc30 con 0x7ff99c0613a0 2021-03-09T09:49:09.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003c990 con 0x7ff98001dc30 2021-03-09T09:49:09.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:09.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:09.107 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:09.099+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4186 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:09.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:09.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:09.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:09.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:09.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:09.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800505a0 con 0x7ff98001dc30 2021-03-09T09:49:09.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800226a0 con 0x7ff98001dc30 2021-03-09T09:49:09.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7ff98004ff20 tx=0x7ff9940739d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:09.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99403b8a0 con 0x7ff98001dc30 2021-03-09T09:49:09.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99403b8a0 con 0x7ff98001dc30 2021-03-09T09:49:09.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994008030 con 0x7ff98001dc30 2021-03-09T09:49:09.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.107+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:09.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:09.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:09.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:14.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:14.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:14.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7ff98004ff20 tx=0x7ff9940739d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:14.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7ff98004ff20 tx=0x7ff9940739d0).stop 2021-03-09T09:49:14.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:14.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:14.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:14.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:14.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800370f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:14.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800450b0 con 0x7ff99c0adc20 2021-03-09T09:49:14.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095930 con 0x7ff99c0613a0 2021-03-09T09:49:14.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dc30 con 0x7ff98003bce0 2021-03-09T09:49:14.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:14.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:14.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:14.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:14.109 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:14.099+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4226 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:14.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:14.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:14.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:14.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:14.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:14.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff98003bce0 2021-03-09T09:49:14.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800505a0 con 0x7ff98003bce0 2021-03-09T09:49:14.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800370f0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7ff98009a0c0 tx=0x7ff994039900).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:14.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff98003bce0 2021-03-09T09:49:14.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a6c0 con 0x7ff98003bce0 2021-03-09T09:49:14.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.115+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99408a490 con 0x7ff98003bce0 2021-03-09T09:49:14.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.115+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:14.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.115+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:14.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:14.115+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:19.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800370f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:19.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800370f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:19.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800370f0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7ff98009a0c0 tx=0x7ff994039900).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:19.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800370f0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7ff98009a0c0 tx=0x7ff994039900).stop 2021-03-09T09:49:19.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800370f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:19.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800370f0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:19.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:19.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:19.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:19.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f770 con 0x7ff99c0adc20 2021-03-09T09:49:19.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:19.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a960 con 0x7ff99c0613a0 2021-03-09T09:49:19.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003c990 con 0x7ff98001dc30 2021-03-09T09:49:19.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:19.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:19.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:19.110 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:19.099+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4260 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:19.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:19.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:19.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:19.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:19.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:19.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980011eb0 con 0x7ff98001dc30 2021-03-09T09:49:19.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff98001dc30 2021-03-09T09:49:19.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff99403ddc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:19.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994073c20 con 0x7ff98001dc30 2021-03-09T09:49:19.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940b6480 con 0x7ff98001dc30 2021-03-09T09:49:19.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994081940 con 0x7ff98001dc30 2021-03-09T09:49:19.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.107+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:19.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:19.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:19.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:24.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:24.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:24.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff99403ddc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:24.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7ff994009d80 tx=0x7ff99403ddc0).stop 2021-03-09T09:49:24.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:24.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:24.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:24.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98004c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:24.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:24.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800450b0 con 0x7ff99c0adc20 2021-03-09T09:49:24.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dc30 con 0x7ff99c0613a0 2021-03-09T09:49:24.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003c990 con 0x7ff98003bce0 2021-03-09T09:49:24.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:24.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:24.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:24.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:24.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98004c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:24.112 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:24.103+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4298 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:24.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:24.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:24.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:24.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:24.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001e790 con 0x7ff98003bce0 2021-03-09T09:49:24.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054c40 con 0x7ff98003bce0 2021-03-09T09:49:24.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98004c3d0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7ff98004ff80 tx=0x7ff98c004ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:24.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c023070 con 0x7ff98003bce0 2021-03-09T09:49:24.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98003bce0 2021-03-09T09:49:24.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98003bce0 2021-03-09T09:49:24.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.111+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:24.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:24.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:24.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:25.844 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:49:25.849 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:49:29.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98004c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:29.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98004c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:29.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98004c3d0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7ff98004ff80 tx=0x7ff98c004ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:29.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98004c3d0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7ff98004ff80 tx=0x7ff98c004ef0).stop 2021-03-09T09:49:29.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98004c3d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:29.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98004c3d0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:29.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:29.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:29.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:29.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800450b0 con 0x7ff99c0adc20 2021-03-09T09:49:29.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017b10 con 0x7ff99c0613a0 2021-03-09T09:49:29.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800973b0 con 0x7ff98001dc30 2021-03-09T09:49:29.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:29.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:29.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:29.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:29.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:29.114 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:29.103+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4330 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:29.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:29.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:29.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:29.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:29.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980043ff0 con 0x7ff98001dc30 2021-03-09T09:49:29.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001e790 con 0x7ff98001dc30 2021-03-09T09:49:29.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7ff98c008cc0 tx=0x7ff98c03cef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:29.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff98001dc30 2021-03-09T09:49:29.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98001dc30 2021-03-09T09:49:29.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98001dc30 2021-03-09T09:49:29.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.111+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:29.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:29.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:29.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:34.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:34.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:34.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7ff98c008cc0 tx=0x7ff98c03cef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:34.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7ff98c008cc0 tx=0x7ff98c03cef0).stop 2021-03-09T09:49:34.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 msgr2=0x7ff980099bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:34.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98001dc30 0x7ff980099bd0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:34.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:34.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:34.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:34.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003c990 con 0x7ff99c0adc20 2021-03-09T09:49:34.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800450b0 con 0x7ff99c0613a0 2021-03-09T09:49:34.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dc30 con 0x7ff98003bce0 2021-03-09T09:49:34.114 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:34.107+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4371 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:34.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:34.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:34.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:34.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:34.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:34.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:34.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:34.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:34.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:34.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000dbb0 con 0x7ff98003bce0 2021-03-09T09:49:34.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980011eb0 con 0x7ff98003bce0 2021-03-09T09:49:34.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051050 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7ff98002a710 tx=0x7ff9940816e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:34.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994039740 con 0x7ff98003bce0 2021-03-09T09:49:34.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994070210 con 0x7ff98003bce0 2021-03-09T09:49:34.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.115+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994031e40 con 0x7ff98003bce0 2021-03-09T09:49:34.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.115+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:34.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.115+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:34.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:34.115+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:39.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980051050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:39.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980051050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:39.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051050 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7ff98002a710 tx=0x7ff9940816e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:39.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051050 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7ff98002a710 tx=0x7ff9940816e0).stop 2021-03-09T09:49:39.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980051050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:39.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051050 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:39.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:39.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:39.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:39.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800450b0 con 0x7ff99c0adc20 2021-03-09T09:49:39.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011ab0 con 0x7ff99c0613a0 2021-03-09T09:49:39.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b000 con 0x7ff980051540 2021-03-09T09:49:39.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:39.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:39.115 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:39.107+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4404 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:39.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:39.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:39.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:39.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.111+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:39.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.111+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:39.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.111+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:39.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.111+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:39.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.111+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004dda0 con 0x7ff980051540 2021-03-09T09:49:39.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.111+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000dbb0 con 0x7ff980051540 2021-03-09T09:49:39.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.111+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7ff994014000 tx=0x7ff99406a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:39.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994040020 con 0x7ff980051540 2021-03-09T09:49:39.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.115+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994073c40 con 0x7ff980051540 2021-03-09T09:49:39.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.119+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:49:39.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.119+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:39.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.119+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:39.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:39.119+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:44.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:44.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:44.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7ff994014000 tx=0x7ff99406a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:44.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7ff994014000 tx=0x7ff99406a000).stop 2021-03-09T09:49:44.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:44.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:44.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:44.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:44.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:44.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800961e0 con 0x7ff99c0adc20 2021-03-09T09:49:44.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017b10 con 0x7ff99c0613a0 2021-03-09T09:49:44.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a9a0 con 0x7ff98003bce0 2021-03-09T09:49:44.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:44.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:44.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:44.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:44.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:44.118 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:44.107+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4442 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:44.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:44.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:44.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:44.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:44.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002a120 con 0x7ff98003bce0 2021-03-09T09:49:44.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004dda0 con 0x7ff98003bce0 2021-03-09T09:49:44.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.111+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800519a0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7ff99408b040 tx=0x7ff994027f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:44.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff98003bce0 2021-03-09T09:49:44.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff98003bce0 2021-03-09T09:49:44.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.115+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98003bce0 2021-03-09T09:49:44.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.115+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:44.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.119+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:44.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:44.119+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:49.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:49:49.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:49.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800519a0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7ff99408b040 tx=0x7ff994027f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:49.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800519a0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7ff99408b040 tx=0x7ff994027f50).stop 2021-03-09T09:49:49.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800519a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:49.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800519a0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:49.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:49.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:49.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:49.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d570 con 0x7ff99c0adc20 2021-03-09T09:49:49.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dc30 con 0x7ff99c0613a0 2021-03-09T09:49:49.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800450b0 con 0x7ff980051540 2021-03-09T09:49:49.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:49.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:49.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:49.118 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:49.110+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4479 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:49.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:49.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:49.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:49.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:49.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:49.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:49.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800505a0 con 0x7ff980051540 2021-03-09T09:49:49.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980042210 con 0x7ff980051540 2021-03-09T09:49:49.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7ff984078010 tx=0x7ff984081360).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:49.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840447a0 con 0x7ff980051540 2021-03-09T09:49:49.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984044920 con 0x7ff980051540 2021-03-09T09:49:49.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984025a40 con 0x7ff980051540 2021-03-09T09:49:49.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.114+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:49.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.114+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:49.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:49.114+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:54.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:54.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:54.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7ff984078010 tx=0x7ff984081360).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:54.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7ff984078010 tx=0x7ff984081360).stop 2021-03-09T09:49:54.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:54.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:54.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:54.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:54.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:54.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800230e0 con 0x7ff99c0adc20 2021-03-09T09:49:54.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097570 con 0x7ff99c0613a0 2021-03-09T09:49:54.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d570 con 0x7ff98003bce0 2021-03-09T09:49:54.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:54.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:54.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:54.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:49:54.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:54.122 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:54.110+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4516 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:54.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:54.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:54.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:54.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:54.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980048ff0 con 0x7ff98003bce0 2021-03-09T09:49:54.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980043ff0 con 0x7ff98003bce0 2021-03-09T09:49:54.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.110+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011bf0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7ff98c008a90 tx=0x7ff98c0a30d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:54.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053080 con 0x7ff98003bce0 2021-03-09T09:49:54.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98003bce0 2021-03-09T09:49:54.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98003bce0 2021-03-09T09:49:54.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.118+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:54.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.122+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:54.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:54.122+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:49:55.891 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:49:55.893 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:49:59.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:59.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:59.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011bf0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7ff98c008a90 tx=0x7ff98c0a30d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:59.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011bf0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7ff98c008a90 tx=0x7ff98c0a30d0).stop 2021-03-09T09:49:59.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:59.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011bf0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:59.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:59.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:59.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:59.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a560 con 0x7ff99c0adc20 2021-03-09T09:49:59.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dc30 con 0x7ff99c0613a0 2021-03-09T09:49:59.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002fa70 con 0x7ff980051540 2021-03-09T09:49:59.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:49:59.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:59.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:59.122 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:49:59.110+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4551 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:49:59.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:59.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:59.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:59.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:59.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000dbb0 con 0x7ff980051540 2021-03-09T09:49:59.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800505a0 con 0x7ff980051540 2021-03-09T09:49:59.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840a6e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:59.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019090 con 0x7ff980051540 2021-03-09T09:49:59.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984000c70 con 0x7ff980051540 2021-03-09T09:49:59.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984037cb0 con 0x7ff980051540 2021-03-09T09:49:59.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.118+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:59.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.118+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:59.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:49:59.118+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:04.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:04.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:04.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840a6e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:04.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840a6e90).stop 2021-03-09T09:50:04.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:04.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:04.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:04.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:04.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:04.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800450b0 con 0x7ff99c0adc20 2021-03-09T09:50:04.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d570 con 0x7ff99c0613a0 2021-03-09T09:50:04.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800139f0 con 0x7ff98003bce0 2021-03-09T09:50:04.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:04.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.110+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:04.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.114+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:04.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.114+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:04.124 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:04.110+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4586 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:04.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:04.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:04.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:04.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:04.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:04.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980090e00 con 0x7ff98003bce0 2021-03-09T09:50:04.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980048ff0 con 0x7ff98003bce0 2021-03-09T09:50:04.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.118+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051920 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7ff98c020d00 tx=0x7ff98c008940).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:04.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff98003bce0 2021-03-09T09:50:04.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff98003bce0 2021-03-09T09:50:04.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c000bf0 con 0x7ff98003bce0 2021-03-09T09:50:04.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.122+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:04.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.122+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:04.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:04.122+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:09.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980051920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:09.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980051920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:09.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051920 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7ff98c020d00 tx=0x7ff98c008940).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:09.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051920 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7ff98c020d00 tx=0x7ff98c008940).stop 2021-03-09T09:50:09.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980051920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:09.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980051920 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:09.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:09.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:09.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:09.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001cac0 con 0x7ff99c0adc20 2021-03-09T09:50:09.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dc30 con 0x7ff99c0613a0 2021-03-09T09:50:09.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000acc0 con 0x7ff980051540 2021-03-09T09:50:09.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:09.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:09.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:09.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:09.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:09.125 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:09.114+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4618 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:09.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:09.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:09.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:09.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:09.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003c940 con 0x7ff980051540 2021-03-09T09:50:09.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980090e00 con 0x7ff980051540 2021-03-09T09:50:09.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051e10 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7ff98c08e890 tx=0x7ff98c0a2f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:09.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980051540 2021-03-09T09:50:09.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e070 con 0x7ff980051540 2021-03-09T09:50:09.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T09:50:09.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.126+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:09.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.130+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:09.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:09.130+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:14.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:14.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:14.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051e10 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7ff98c08e890 tx=0x7ff98c0a2f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:14.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051e10 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7ff98c08e890 tx=0x7ff98c0a2f40).stop 2021-03-09T09:50:14.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:14.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051e10 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:14.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98002cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:14.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:14.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:14.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d570 con 0x7ff99c0adc20 2021-03-09T09:50:14.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dc30 con 0x7ff99c0613a0 2021-03-09T09:50:14.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038f60 con 0x7ff98003bce0 2021-03-09T09:50:14.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:14.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:14.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98002cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:14.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:14.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.114+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:14.127 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:14.114+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4656 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:14.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:14.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:14.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:14.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:14.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008d3e0 con 0x7ff98003bce0 2021-03-09T09:50:14.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000dbb0 con 0x7ff98003bce0 2021-03-09T09:50:14.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98002cb60 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7ff98402cab0 tx=0x7ff9840583f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:14.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984084a00 con 0x7ff98003bce0 2021-03-09T09:50:14.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984084c30 con 0x7ff98003bce0 2021-03-09T09:50:14.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400f1e0 con 0x7ff98003bce0 2021-03-09T09:50:14.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.122+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:14.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.126+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:14.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:14.126+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:19.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98002cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:19.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98002cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:19.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98002cb60 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7ff98402cab0 tx=0x7ff9840583f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:19.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98002cb60 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7ff98402cab0 tx=0x7ff9840583f0).stop 2021-03-09T09:50:19.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98002cb60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:19.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98002cb60 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:19.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:19.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:19.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:19.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000acc0 con 0x7ff99c0adc20 2021-03-09T09:50:19.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800450b0 con 0x7ff99c0613a0 2021-03-09T09:50:19.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d570 con 0x7ff980051540 2021-03-09T09:50:19.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:19.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:19.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:19.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:19.129 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:19.118+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4695 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:19.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:19.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:19.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:19.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:19.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:19.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:50:19.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008d3e0 con 0x7ff980051540 2021-03-09T09:50:19.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.122+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005bf0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840a6020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:19.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019080 con 0x7ff980051540 2021-03-09T09:50:19.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984006aa0 con 0x7ff980051540 2021-03-09T09:50:19.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984005b70 con 0x7ff980051540 2021-03-09T09:50:19.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.130+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:19.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.130+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:19.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:19.130+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:24.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:24.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:24.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005bf0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840a6020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:24.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005bf0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff9840a6020).stop 2021-03-09T09:50:24.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:24.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005bf0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:24.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:24.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:24.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:24.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017b10 con 0x7ff99c0adc20 2021-03-09T09:50:24.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003e390 con 0x7ff99c0613a0 2021-03-09T09:50:24.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038170 con 0x7ff98003bce0 2021-03-09T09:50:24.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:24.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:24.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:24.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:24.129 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:24.118+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4734 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:24.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.118+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:24.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.122+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:24.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.122+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:24.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.122+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:24.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.122+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:24.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.122+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980018510 con 0x7ff98003bce0 2021-03-09T09:50:24.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.122+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff98003bce0 2021-03-09T09:50:24.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.122+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010ca0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7ff984085b60 tx=0x7ff984085a10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:24.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984037930 con 0x7ff98003bce0 2021-03-09T09:50:24.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984037930 con 0x7ff98003bce0 2021-03-09T09:50:24.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984036e50 con 0x7ff98003bce0 2021-03-09T09:50:24.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.130+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:24.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.130+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:24.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:24.130+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:26.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:50:26.015 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:50:29.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980010ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:29.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980010ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:29.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010ca0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7ff984085b60 tx=0x7ff984085a10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:29.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010ca0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7ff984085b60 tx=0x7ff984085a10).stop 2021-03-09T09:50:29.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980010ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:29.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010ca0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:29.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:29.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:29.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:29.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044040 con 0x7ff99c0adc20 2021-03-09T09:50:29.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dc30 con 0x7ff99c0613a0 2021-03-09T09:50:29.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090700 con 0x7ff980051540 2021-03-09T09:50:29.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:29.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:29.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:29.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:29.131 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:29.122+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4769 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:29.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:29.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:29.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:29.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:29.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:29.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001da40 con 0x7ff980051540 2021-03-09T09:50:29.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002a120 con 0x7ff980051540 2021-03-09T09:50:29.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005570 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7ff980023840 tx=0x7ff994027eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:29.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400bac0 con 0x7ff980051540 2021-03-09T09:50:29.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400bac0 con 0x7ff980051540 2021-03-09T09:50:29.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:50:29.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.134+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:29.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.134+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:29.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:29.134+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:34.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:50:34.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:34.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005570 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7ff980023840 tx=0x7ff994027eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:34.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005570 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7ff980023840 tx=0x7ff994027eb0).stop 2021-03-09T09:50:34.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:34.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005570 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:34.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:34.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:34.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:34.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800197c0 con 0x7ff99c0adc20 2021-03-09T09:50:34.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dc30 con 0x7ff99c0613a0 2021-03-09T09:50:34.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f300 con 0x7ff98003bce0 2021-03-09T09:50:34.132 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:34.122+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4802 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:34.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:34.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:34.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:34.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:34.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:34.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:34.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:34.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:34.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:34.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800172e0 con 0x7ff98003bce0 2021-03-09T09:50:34.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003c940 con 0x7ff98003bce0 2021-03-09T09:50:34.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011160 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7ff98c03d300 tx=0x7ff98c00baf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:34.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff98003bce0 2021-03-09T09:50:34.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff98003bce0 2021-03-09T09:50:34.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c051da0 con 0x7ff98003bce0 2021-03-09T09:50:34.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.130+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:34.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.130+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:34.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:34.130+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:39.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:39.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:39.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011160 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7ff98c03d300 tx=0x7ff98c00baf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:39.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011160 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7ff98c03d300 tx=0x7ff98c00baf0).stop 2021-03-09T09:50:39.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:39.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011160 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:39.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:39.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:39.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980014bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:39.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a400 con 0x7ff99c0adc20 2021-03-09T09:50:39.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e150 con 0x7ff99c0613a0 2021-03-09T09:50:39.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048200 con 0x7ff980051540 2021-03-09T09:50:39.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:39.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:39.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:39.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:39.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980014bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:39.135 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:39.126+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4834 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:39.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:39.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:39.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:39.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:39.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d2e0 con 0x7ff980051540 2021-03-09T09:50:39.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001da40 con 0x7ff980051540 2021-03-09T09:50:39.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980014bc0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7ff98004c2d0 tx=0x7ff994036c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:39.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994040080 con 0x7ff980051540 2021-03-09T09:50:39.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980051540 2021-03-09T09:50:39.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980051540 2021-03-09T09:50:39.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.134+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:39.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.134+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:39.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:39.134+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:44.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980014bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:50:44.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980014bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:44.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980014bc0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7ff98004c2d0 tx=0x7ff994036c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:44.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980014bc0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7ff98004c2d0 tx=0x7ff994036c60).stop 2021-03-09T09:50:44.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980014bc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:44.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980014bc0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:44.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:44.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:44.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:44.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a960 con 0x7ff99c0adc20 2021-03-09T09:50:44.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003cb90 con 0x7ff99c0613a0 2021-03-09T09:50:44.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027a90 con 0x7ff98003bce0 2021-03-09T09:50:44.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:44.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:44.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:44.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.122+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:44.135 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:44.122+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4874 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:44.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.126+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:44.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.126+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:44.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.126+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:44.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.126+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:44.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.126+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:44.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.126+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff98003bce0 2021-03-09T09:50:44.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.126+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800172e0 con 0x7ff98003bce0 2021-03-09T09:50:44.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.126+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7ff98c08ea70 tx=0x7ff98c079700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:44.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff98003bce0 2021-03-09T09:50:44.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff98003bce0 2021-03-09T09:50:44.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98003bce0 2021-03-09T09:50:44.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.134+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:44.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.134+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:44.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:44.134+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:49.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:50:49.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:49.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7ff98c08ea70 tx=0x7ff98c079700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:49.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7ff98c08ea70 tx=0x7ff98c079700).stop 2021-03-09T09:50:49.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:49.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:49.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:49.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:49.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:49.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014300 con 0x7ff99c0adc20 2021-03-09T09:50:49.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026710 con 0x7ff99c0613a0 2021-03-09T09:50:49.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027a90 con 0x7ff980051540 2021-03-09T09:50:49.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:49.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:49.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:49.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:49.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.126+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:49.137 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:49.126+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4907 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:49.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.130+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:49.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.130+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:49.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.130+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:49.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.130+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:49.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.130+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017150 con 0x7ff980051540 2021-03-09T09:50:49.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.130+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:50:49.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.130+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7ff98c07dcd0 tx=0x7ff98c079320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:49.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a1040 con 0x7ff980051540 2021-03-09T09:50:49.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d020 con 0x7ff980051540 2021-03-09T09:50:49.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980051540 2021-03-09T09:50:49.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.138+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:49.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.138+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:49.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:49.138+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:54.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:50:54.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:54.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7ff98c07dcd0 tx=0x7ff98c079320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:54.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7ff98c07dcd0 tx=0x7ff98c079320).stop 2021-03-09T09:50:54.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980099bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:54.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980099bd0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:54.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:54.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:54.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980018ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:54.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800165f0 con 0x7ff99c0adc20 2021-03-09T09:50:54.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023700 con 0x7ff99c0613a0 2021-03-09T09:50:54.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044240 con 0x7ff98003bce0 2021-03-09T09:50:54.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:54.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:54.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:54.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:54.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980018ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:54.144 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:54.130+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4944 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:54.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:54.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:54.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:54.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:54.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003e380 con 0x7ff98003bce0 2021-03-09T09:50:54.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d2e0 con 0x7ff98003bce0 2021-03-09T09:50:54.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980018ec0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff994070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:54.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff98003bce0 2021-03-09T09:50:54.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff98003bce0 2021-03-09T09:50:54.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.138+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff98003bce0 2021-03-09T09:50:54.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.138+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:54.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.138+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:54.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:54.138+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:50:56.059 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:50:56.062 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:50:59.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980018ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:59.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980018ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:59.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980018ec0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff994070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:59.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980018ec0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff994070040).stop 2021-03-09T09:50:59.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980018ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:59.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980018ec0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:59.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:59.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:59.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:50:59.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:59.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:59.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023700 con 0x7ff99c0adc20 2021-03-09T09:50:59.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004c930 con 0x7ff99c0613a0 2021-03-09T09:50:59.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052ec0 con 0x7ff980051540 2021-03-09T09:50:59.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:50:59.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:59.140 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:50:59.129+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4980 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:50:59.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:59.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:59.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:59.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:59.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:59.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001c520 con 0x7ff980051540 2021-03-09T09:50:59.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003e380 con 0x7ff980051540 2021-03-09T09:50:59.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.129+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051920 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7ff99408c360 tx=0x7ff994031f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:59.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99403f130 con 0x7ff980051540 2021-03-09T09:50:59.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99403f290 con 0x7ff980051540 2021-03-09T09:50:59.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.137+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940a9ba0 con 0x7ff980051540 2021-03-09T09:50:59.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.137+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:59.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.137+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:59.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:50:59.137+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:04.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:04.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:04.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051920 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7ff99408c360 tx=0x7ff994031f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:04.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051920 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7ff99408c360 tx=0x7ff994031f20).stop 2021-03-09T09:51:04.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:04.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051920 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:04.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:04.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:04.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:04.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0adc20 2021-03-09T09:51:04.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053060 con 0x7ff99c0613a0 2021-03-09T09:51:04.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004d70 con 0x7ff98003bce0 2021-03-09T09:51:04.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:04.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:04.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:04.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:04.142 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:04.133+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5021 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:04.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:04.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:04.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:04.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:04.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:04.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002dc30 con 0x7ff98003bce0 2021-03-09T09:51:04.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017150 con 0x7ff98003bce0 2021-03-09T09:51:04.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.133+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7ff98c090000 tx=0x7ff98c0a2d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:04.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.137+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a1080 con 0x7ff98003bce0 2021-03-09T09:51:04.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d020 con 0x7ff98003bce0 2021-03-09T09:51:04.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98003bce0 2021-03-09T09:51:04.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.141+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:04.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.141+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:04.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:04.141+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:09.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:09.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:09.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7ff98c090000 tx=0x7ff98c0a2d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:09.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7ff98c090000 tx=0x7ff98c0a2d20).stop 2021-03-09T09:51:09.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:09.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020ff0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:09.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800051b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:09.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:09.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:09.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a600 con 0x7ff99c0adc20 2021-03-09T09:51:09.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0613a0 2021-03-09T09:51:09.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003c5e0 con 0x7ff980051540 2021-03-09T09:51:09.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:09.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:09.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:09.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:09.149 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:09.133+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5057 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:09.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:09.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:09.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:09.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:09.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:09.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:51:09.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980018510 con 0x7ff980051540 2021-03-09T09:51:09.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.133+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800051b0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7ff98407b120 tx=0x7ff9840813a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:09.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984027020 con 0x7ff980051540 2021-03-09T09:51:09.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840584e0 con 0x7ff980051540 2021-03-09T09:51:09.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980051540 2021-03-09T09:51:09.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.141+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:09.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.141+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:09.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:09.141+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:14.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:14.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:14.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800051b0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7ff98407b120 tx=0x7ff9840813a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:14.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800051b0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7ff98407b120 tx=0x7ff9840813a0).stop 2021-03-09T09:51:14.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800051b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:14.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800051b0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:14.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:14.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:14.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:14.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0adc20 2021-03-09T09:51:14.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f7e0 con 0x7ff99c0613a0 2021-03-09T09:51:14.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038e10 con 0x7ff98003bce0 2021-03-09T09:51:14.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:14.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:14.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:14.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:14.145 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:14.133+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5089 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:14.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.133+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:14.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.137+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:14.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.137+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:14.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.137+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:14.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.137+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:14.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.137+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800503f0 con 0x7ff98003bce0 2021-03-09T09:51:14.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.137+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff98003bce0 2021-03-09T09:51:14.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.141+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020c40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7ff984003b10 tx=0x7ff984044260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:14.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984033170 con 0x7ff98003bce0 2021-03-09T09:51:14.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840582d0 con 0x7ff98003bce0 2021-03-09T09:51:14.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984033170 con 0x7ff98003bce0 2021-03-09T09:51:14.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.145+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:14.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.145+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:14.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:14.145+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:19.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:19.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:19.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020c40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7ff984003b10 tx=0x7ff984044260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:19.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020c40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7ff984003b10 tx=0x7ff984044260).stop 2021-03-09T09:51:19.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:19.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020c40 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:19.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:19.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:19.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:19.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0adc20 2021-03-09T09:51:19.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:19.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:19.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a920 con 0x7ff99c0613a0 2021-03-09T09:51:19.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b980 con 0x7ff980051540 2021-03-09T09:51:19.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:19.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:19.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.137+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:19.147 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:19.137+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5124 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:19.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.141+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:19.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.141+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:19.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.141+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:19.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.141+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:19.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.141+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800303f0 con 0x7ff980051540 2021-03-09T09:51:19.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.141+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001c520 con 0x7ff980051540 2021-03-09T09:51:19.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.141+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052020 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7ff9940816f0 tx=0x7ff9940114e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:19.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T09:51:19.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a250 con 0x7ff980051540 2021-03-09T09:51:19.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:51:19.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.149+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:19.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.149+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:19.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:19.149+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:24.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980052020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:24.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980052020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:24.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052020 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7ff9940816f0 tx=0x7ff9940114e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:24.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052020 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7ff9940816f0 tx=0x7ff9940114e0).stop 2021-03-09T09:51:24.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980052020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:24.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052020 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:24.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:24.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:24.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:24.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0adc20 2021-03-09T09:51:24.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ef10 con 0x7ff99c0613a0 2021-03-09T09:51:24.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980032ab0 con 0x7ff98003bce0 2021-03-09T09:51:24.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:24.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:24.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:24.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:24.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.137+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:24.148 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:24.141+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5163 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:24.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.141+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:24.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:24.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.141+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:24.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:24.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.141+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff98003bce0 2021-03-09T09:51:24.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.141+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002dc30 con 0x7ff98003bce0 2021-03-09T09:51:24.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7ff98c0792f0 tx=0x7ff98c04c1c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:24.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a1040 con 0x7ff98003bce0 2021-03-09T09:51:24.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff98003bce0 2021-03-09T09:51:24.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.153+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98003bce0 2021-03-09T09:51:24.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.153+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:24.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.153+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:24.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:24.153+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:26.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:51:26.104 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:51:29.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:29.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:29.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7ff98c0792f0 tx=0x7ff98c04c1c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:29.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7ff98c0792f0 tx=0x7ff98c04c1c0).stop 2021-03-09T09:51:29.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:29.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:29.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:29.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:29.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:29.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980032ab0 con 0x7ff99c0adc20 2021-03-09T09:51:29.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0613a0 2021-03-09T09:51:29.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018a70 con 0x7ff980051540 2021-03-09T09:51:29.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:29.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:29.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:29.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:29.149 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:29.141+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5198 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:29.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:29.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:29.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.145+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:29.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:29.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.145+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:29.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980004d30 con 0x7ff980051540 2021-03-09T09:51:29.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:51:29.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.145+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 rx=0x7ff9800156d0 tx=0x7ff98c099d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:29.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a1070 con 0x7ff980051540 2021-03-09T09:51:29.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04d040 con 0x7ff980051540 2021-03-09T09:51:29.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.153+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T09:51:29.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.153+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:29.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.153+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:29.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:29.153+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:34.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:34.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:34.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 rx=0x7ff9800156d0 tx=0x7ff98c099d50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:34.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 rx=0x7ff9800156d0 tx=0x7ff98c099d50).stop 2021-03-09T09:51:34.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:34.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:34.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:34.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980052020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:34.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:34.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0adc20 2021-03-09T09:51:34.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013670 con 0x7ff99c0613a0 2021-03-09T09:51:34.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800083b0 con 0x7ff98003bce0 2021-03-09T09:51:34.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:34.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:34.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:34.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:34.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980052020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:34.151 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:34.141+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5229 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:34.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:34.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.145+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:34.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:34.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.145+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:34.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002ce30 con 0x7ff98003bce0 2021-03-09T09:51:34.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980004d30 con 0x7ff98003bce0 2021-03-09T09:51:34.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.145+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980052020 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c003ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:34.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053020 con 0x7ff98003bce0 2021-03-09T09:51:34.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c099ec0 con 0x7ff98003bce0 2021-03-09T09:51:34.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.153+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff98003bce0 2021-03-09T09:51:34.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.153+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:34.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.153+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:34.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:34.153+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:39.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.141+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980052020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:51:39.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.141+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980052020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:39.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980052020 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c003ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:39.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.141+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980052020 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c003ef0).stop 2021-03-09T09:51:39.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.141+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980052020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:39.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.141+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980052020 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:39.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.141+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:39.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:39.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:39.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800437c0 con 0x7ff99c0adc20 2021-03-09T09:51:39.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011a40 con 0x7ff99c0613a0 2021-03-09T09:51:39.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001c040 con 0x7ff980051540 2021-03-09T09:51:39.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:39.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:39.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:39.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:39.152 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:39.145+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5267 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:39.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:39.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:39.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:39.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:39.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:39.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002f460 con 0x7ff980051540 2021-03-09T09:51:39.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.145+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800303f0 con 0x7ff980051540 2021-03-09T09:51:39.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.149+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cbd0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7ff994001000 tx=0x7ff9940071c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:39.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T09:51:39.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994030740 con 0x7ff980051540 2021-03-09T09:51:39.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.157+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980051540 2021-03-09T09:51:39.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.157+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:39.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.157+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:39.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:39.157+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:44.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:44.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:44.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cbd0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7ff994001000 tx=0x7ff9940071c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:44.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cbd0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7ff994001000 tx=0x7ff9940071c0).stop 2021-03-09T09:51:44.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008cbd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:44.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cbd0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:44.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:44.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:44.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:44.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800944c0 con 0x7ff99c0adc20 2021-03-09T09:51:44.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000acc0 con 0x7ff99c0613a0 2021-03-09T09:51:44.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff98003bce0 2021-03-09T09:51:44.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:44.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:44.154 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:44.145+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5304 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:44.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:44.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:44.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.145+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:44.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.149+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:44.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.149+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:44.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.149+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:44.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.149+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:44.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.149+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff98003bce0 2021-03-09T09:51:44.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.149+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002ce30 con 0x7ff98003bce0 2021-03-09T09:51:44.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.153+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134b0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 rx=0x7ff98c03dcd0 tx=0x7ff98c00ba80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:44.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.153+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053030 con 0x7ff98003bce0 2021-03-09T09:51:44.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.153+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d080 con 0x7ff98003bce0 2021-03-09T09:51:44.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.157+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98003bce0 2021-03-09T09:51:44.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.157+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:44.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.157+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:44.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:44.157+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:49.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800134b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:49.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800134b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:49.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134b0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 rx=0x7ff98c03dcd0 tx=0x7ff98c00ba80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:49.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134b0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 rx=0x7ff98c03dcd0 tx=0x7ff98c00ba80).stop 2021-03-09T09:51:49.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800134b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:49.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134b0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:49.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:49.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:49.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:49.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004d80 con 0x7ff99c0adc20 2021-03-09T09:51:49.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0613a0 2021-03-09T09:51:49.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.145+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800389b0 con 0x7ff980051540 2021-03-09T09:51:49.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:49.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:49.155 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:49.149+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5344 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:49.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:49.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:49.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:49.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:49.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:49.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:49.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:49.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001e930 con 0x7ff980051540 2021-03-09T09:51:49.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002f460 con 0x7ff980051540 2021-03-09T09:51:49.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.149+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bd0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 rx=0x7ff9940b6060 tx=0x7ff99408bf70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:49.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.153+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T09:51:49.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.153+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99403d810 con 0x7ff980051540 2021-03-09T09:51:49.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.157+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980051540 2021-03-09T09:51:49.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.157+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:49.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.157+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:49.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:49.157+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:54.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980020bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:54.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980020bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:54.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bd0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 rx=0x7ff9940b6060 tx=0x7ff99408bf70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:54.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bd0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 rx=0x7ff9940b6060 tx=0x7ff99408bf70).stop 2021-03-09T09:51:54.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980020bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:54.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bd0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:54.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:54.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:54.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:54.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011a40 con 0x7ff99c0adc20 2021-03-09T09:51:54.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012070 con 0x7ff99c0613a0 2021-03-09T09:51:54.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004d80 con 0x7ff98003bce0 2021-03-09T09:51:54.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:54.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:54.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:54.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:54.157 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:54.149+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5380 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:54.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:54.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:54.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:54.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:54.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:54.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001ac70 con 0x7ff98003bce0 2021-03-09T09:51:54.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.149+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800503f0 con 0x7ff98003bce0 2021-03-09T09:51:54.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.153+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012f70 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 rx=0x7ff984009010 tx=0x7ff98405d160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:54.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.153+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984000cd0 con 0x7ff98003bce0 2021-03-09T09:51:54.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.153+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff98003bce0 2021-03-09T09:51:54.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.157+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff98003bce0 2021-03-09T09:51:54.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.157+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:54.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.161+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:54.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:54.161+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:51:56.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:51:56.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:51:59.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980012f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:59.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980012f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:59.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012f70 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 rx=0x7ff984009010 tx=0x7ff98405d160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:59.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012f70 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 rx=0x7ff984009010 tx=0x7ff98405d160).stop 2021-03-09T09:51:59.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980012f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:59.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012f70 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:59.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:59.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:59.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:59.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012070 con 0x7ff99c0adc20 2021-03-09T09:51:59.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004d80 con 0x7ff99c0613a0 2021-03-09T09:51:59.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800437c0 con 0x7ff980051540 2021-03-09T09:51:59.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:51:59.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:51:59.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:59.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:59.159 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:51:59.149+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5411 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:51:59.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.149+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:59.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.153+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:59.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.153+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:59.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.153+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:59.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.153+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:59.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.153+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980046550 con 0x7ff980051540 2021-03-09T09:51:59.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.153+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001e930 con 0x7ff980051540 2021-03-09T09:51:59.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.153+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003de20 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff9940701e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:59.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.157+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994040030 con 0x7ff980051540 2021-03-09T09:51:59.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.157+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994066a40 con 0x7ff980051540 2021-03-09T09:51:59.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.161+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:51:59.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.161+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:59.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.161+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:59.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:51:59.161+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:04.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:04.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:04.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003de20 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff9940701e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:04.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003de20 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff9940701e0).stop 2021-03-09T09:52:04.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003de20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:04.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003de20 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:04.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:04.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:04.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:04.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0adc20 2021-03-09T09:52:04.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800437c0 con 0x7ff99c0613a0 2021-03-09T09:52:04.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018a70 con 0x7ff98003bce0 2021-03-09T09:52:04.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:04.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:04.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.148+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:04.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.152+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:04.161 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:04.152+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5451 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:04.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:04.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:04.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:04.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:04.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:04.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980042b90 con 0x7ff98003bce0 2021-03-09T09:52:04.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980046550 con 0x7ff98003bce0 2021-03-09T09:52:04.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff994009b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:04.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994040070 con 0x7ff98003bce0 2021-03-09T09:52:04.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940ab080 con 0x7ff98003bce0 2021-03-09T09:52:04.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940ab080 con 0x7ff98003bce0 2021-03-09T09:52:04.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.160+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:04.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.160+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:04.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:04.160+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:09.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:09.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:09.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff994009b70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:09.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff994009b70).stop 2021-03-09T09:52:09.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:09.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:09.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:09.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:09.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:09.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008a1d0 con 0x7ff99c0adc20 2021-03-09T09:52:09.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0613a0 2021-03-09T09:52:09.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012070 con 0x7ff980051540 2021-03-09T09:52:09.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:09.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:09.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:09.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:09.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:09.162 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:09.152+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5486 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:09.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:09.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:09.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:09.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:09.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004c600 con 0x7ff980051540 2021-03-09T09:52:09.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.152+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:52:09.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.156+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cba0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c016130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:09.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980051540 2021-03-09T09:52:09.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980051540 2021-03-09T09:52:09.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T09:52:09.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.160+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:09.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.160+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:09.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:09.160+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:14.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:14.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:14.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cba0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c016130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:14.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cba0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c016130).stop 2021-03-09T09:52:14.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008cba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:14.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008cba0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:14.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:14.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:14.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:14.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d0c0 con 0x7ff99c0adc20 2021-03-09T09:52:14.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800437c0 con 0x7ff99c0613a0 2021-03-09T09:52:14.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093940 con 0x7ff98003bce0 2021-03-09T09:52:14.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:14.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:14.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:14.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:14.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.152+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:14.164 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:14.152+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5518 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:14.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:14.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:14.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:14.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:14.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004f900 con 0x7ff98003bce0 2021-03-09T09:52:14.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001ac70 con 0x7ff98003bce0 2021-03-09T09:52:14.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984058340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:14.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840814a0 con 0x7ff98003bce0 2021-03-09T09:52:14.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400b4b0 con 0x7ff98003bce0 2021-03-09T09:52:14.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984076b90 con 0x7ff98003bce0 2021-03-09T09:52:14.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.164+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:14.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.164+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:14.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:14.164+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:19.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:19.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:19.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984058340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:19.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984058340).stop 2021-03-09T09:52:19.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:19.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bd0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:19.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:19.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:19.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:19.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093fc0 con 0x7ff99c0adc20 2021-03-09T09:52:19.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800558b0 con 0x7ff99c0613a0 2021-03-09T09:52:19.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018a70 con 0x7ff980051540 2021-03-09T09:52:19.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:19.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:19.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:19.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:19.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:19.165 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:19.156+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5559 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:19.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:19.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:19.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:19.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:19.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:52:19.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.156+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004c600 con 0x7ff980051540 2021-03-09T09:52:19.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.160+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012440 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7ff98c099b10 tx=0x7ff98c07df20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:19.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff980051540 2021-03-09T09:52:19.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04d040 con 0x7ff980051540 2021-03-09T09:52:19.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff980051540 2021-03-09T09:52:19.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.164+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:19.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.168+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:19.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:19.168+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:24.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:24.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:24.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012440 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7ff98c099b10 tx=0x7ff98c07df20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:24.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012440 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7ff98c099b10 tx=0x7ff98c07df20).stop 2021-03-09T09:52:24.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:24.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012440 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:24.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:24.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:24.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:24.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980007140 con 0x7ff99c0adc20 2021-03-09T09:52:24.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800437c0 con 0x7ff99c0613a0 2021-03-09T09:52:24.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dfd0 con 0x7ff98003bce0 2021-03-09T09:52:24.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:24.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:24.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:24.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:24.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:24.168 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:24.156+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5592 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:24.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:24.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:24.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:24.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:24.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000b640 con 0x7ff98003bce0 2021-03-09T09:52:24.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004f900 con 0x7ff98003bce0 2021-03-09T09:52:24.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.160+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134d0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 rx=0x7ff984005970 tx=0x7ff9840271b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:24.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840813d0 con 0x7ff98003bce0 2021-03-09T09:52:24.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984005b70 con 0x7ff98003bce0 2021-03-09T09:52:24.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984005ce0 con 0x7ff98003bce0 2021-03-09T09:52:24.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.164+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:24.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.164+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:24.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:24.164+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:26.296 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:52:26.304 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:52:29.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800134d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:29.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800134d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:29.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134d0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 rx=0x7ff984005970 tx=0x7ff9840271b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:29.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134d0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 rx=0x7ff984005970 tx=0x7ff9840271b0).stop 2021-03-09T09:52:29.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800134d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:29.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800134d0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:29.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:29.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:29.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:29.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800403b0 con 0x7ff99c0adc20 2021-03-09T09:52:29.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029ba0 con 0x7ff99c0613a0 2021-03-09T09:52:29.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ac80 con 0x7ff980051540 2021-03-09T09:52:29.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:29.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:29.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:29.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:29.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:29.169 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:29.160+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5631 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:29.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:29.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:29.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:29.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:29.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800429c0 con 0x7ff980051540 2021-03-09T09:52:29.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000b640 con 0x7ff980051540 2021-03-09T09:52:29.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.160+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984040dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:29.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984005ed0 con 0x7ff980051540 2021-03-09T09:52:29.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984006aa0 con 0x7ff980051540 2021-03-09T09:52:29.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984076ac0 con 0x7ff980051540 2021-03-09T09:52:29.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.168+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:29.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.172+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:29.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:29.172+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:34.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980020bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:34.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980020bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:34.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984040dc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:34.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984040dc0).stop 2021-03-09T09:52:34.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980020bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:34.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:34.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:34.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:34.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:34.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038cf0 con 0x7ff99c0adc20 2021-03-09T09:52:34.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980089b10 con 0x7ff99c0613a0 2021-03-09T09:52:34.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010ca0 con 0x7ff98003bce0 2021-03-09T09:52:34.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:34.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:34.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:34.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:34.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.160+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:34.170 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:34.160+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5668 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:34.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.164+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:34.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.164+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:34.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.164+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:34.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.164+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:34.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.164+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000edb0 con 0x7ff98003bce0 2021-03-09T09:52:34.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.164+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800429c0 con 0x7ff98003bce0 2021-03-09T09:52:34.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.164+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7ff9840355e0 tx=0x7ff984087160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:34.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98400f180 con 0x7ff98003bce0 2021-03-09T09:52:34.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984005460 con 0x7ff98003bce0 2021-03-09T09:52:34.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400f180 con 0x7ff98003bce0 2021-03-09T09:52:34.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.172+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:34.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.172+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:34.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:34.172+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:39.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:39.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:39.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7ff9840355e0 tx=0x7ff984087160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:39.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7ff9840355e0 tx=0x7ff984087160).stop 2021-03-09T09:52:39.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:39.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:39.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:39.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:39.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:39.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090350 con 0x7ff99c0adc20 2021-03-09T09:52:39.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:39.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011f10 con 0x7ff99c0613a0 2021-03-09T09:52:39.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010ca0 con 0x7ff980051540 2021-03-09T09:52:39.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:39.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:39.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:39.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:39.173 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:39.164+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5704 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:39.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:39.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:39.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:39.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:39.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800150c0 con 0x7ff980051540 2021-03-09T09:52:39.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:52:39.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.164+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003da60 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7ff98c078000 tx=0x7ff98c0066d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:39.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980051540 2021-03-09T09:52:39.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff980051540 2021-03-09T09:52:39.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T09:52:39.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.172+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:39.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.172+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:39.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:39.172+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:44.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:44.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:44.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003da60 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7ff98c078000 tx=0x7ff98c0066d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:44.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003da60 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7ff98c078000 tx=0x7ff98c0066d0).stop 2021-03-09T09:52:44.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003da60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:44.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003da60 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:44.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:44.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:44.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:44.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f470 con 0x7ff99c0adc20 2021-03-09T09:52:44.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800283d0 con 0x7ff99c0613a0 2021-03-09T09:52:44.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980042a50 con 0x7ff98003bce0 2021-03-09T09:52:44.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:44.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:44.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:44.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:44.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:44.174 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:44.164+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5740 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:44.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:44.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:44.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:44.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:44.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980037430 con 0x7ff98003bce0 2021-03-09T09:52:44.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980042b90 con 0x7ff98003bce0 2021-03-09T09:52:44.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.168+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bb0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7ff994006cf0 tx=0x7ff994006fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:44.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff98003bce0 2021-03-09T09:52:44.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081410 con 0x7ff98003bce0 2021-03-09T09:52:44.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994014d90 con 0x7ff98003bce0 2021-03-09T09:52:44.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.172+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:44.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.176+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:44.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:44.176+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:49.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:52:49.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:49.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bb0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7ff994006cf0 tx=0x7ff994006fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:49.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bb0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7ff994006cf0 tx=0x7ff994006fd0).stop 2021-03-09T09:52:49.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980020bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:49.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980020bb0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:49.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:49.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:49.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:49.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097340 con 0x7ff99c0adc20 2021-03-09T09:52:49.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001afe0 con 0x7ff99c0613a0 2021-03-09T09:52:49.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800233f0 con 0x7ff980051540 2021-03-09T09:52:49.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:49.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:49.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:49.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:49.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.164+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:49.176 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:49.164+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5769 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:49.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.168+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:49.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.168+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:49.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.168+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:49.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.168+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:49.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.168+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:52:49.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.168+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800150c0 con 0x7ff980051540 2021-03-09T09:52:49.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.168+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b6f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c045f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:49.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a1070 con 0x7ff980051540 2021-03-09T09:52:49.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff980051540 2021-03-09T09:52:49.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T09:52:49.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.176+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:49.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.176+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:49.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:49.176+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:54.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98001b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:54.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98001b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:54.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b6f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c045f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:54.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b6f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c045f40).stop 2021-03-09T09:52:54.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98001b6f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:54.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b6f0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:54.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980089d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:54.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:54.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:54.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800233f0 con 0x7ff99c0adc20 2021-03-09T09:52:54.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800099a0 con 0x7ff99c0613a0 2021-03-09T09:52:54.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800537e0 con 0x7ff98003bce0 2021-03-09T09:52:54.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:54.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:54.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:54.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:54.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980089d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:54.177 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:54.168+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5812 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:54.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:54.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:54.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:54.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:54.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096070 con 0x7ff98003bce0 2021-03-09T09:52:54.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000edb0 con 0x7ff98003bce0 2021-03-09T09:52:54.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.168+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980089d60 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98400f260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:54.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840371f0 con 0x7ff98003bce0 2021-03-09T09:52:54.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984058460 con 0x7ff98003bce0 2021-03-09T09:52:54.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984035950 con 0x7ff98003bce0 2021-03-09T09:52:54.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.176+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:54.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.176+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:54.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:54.176+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:52:56.369 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:52:56.372 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:52:59.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980089d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:59.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980089d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:59.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980089d60 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98400f260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:59.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980089d60 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98400f260).stop 2021-03-09T09:52:59.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980089d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:59.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980089d60 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:59.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:59.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:59.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:59.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097340 con 0x7ff99c0adc20 2021-03-09T09:52:59.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022130 con 0x7ff99c0613a0 2021-03-09T09:52:59.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014d60 con 0x7ff980051540 2021-03-09T09:52:59.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:52:59.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:59.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:52:59.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:59.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.168+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:59.179 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:52:59.168+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5849 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:52:59.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.172+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:59.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.172+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:59.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.172+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:59.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.172+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:59.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.172+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980032ac0 con 0x7ff980051540 2021-03-09T09:52:59.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.172+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096070 con 0x7ff980051540 2021-03-09T09:52:59.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.172+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840897d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:59.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840270d0 con 0x7ff980051540 2021-03-09T09:52:59.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984035570 con 0x7ff980051540 2021-03-09T09:52:59.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.180+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980051540 2021-03-09T09:52:59.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.180+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:59.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.180+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:59.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:52:59.180+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:04.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.168+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980020bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:04.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.168+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980020bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:04.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.168+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840897d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:04.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.168+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840897d0).stop 2021-03-09T09:53:04.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980020bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:04.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980020bb0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:04.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:04.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.168+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:04.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:04.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:04.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800112c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:04.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:04.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ac80 con 0x7ff99c0adc20 2021-03-09T09:53:04.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f5b0 con 0x7ff99c0613a0 2021-03-09T09:53:04.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:04.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001ba50 con 0x7ff98003bce0 2021-03-09T09:53:04.180 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:04.172+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5880 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:04.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:04.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:04.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:04.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:04.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:04.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800416c0 con 0x7ff98003bce0 2021-03-09T09:53:04.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980037430 con 0x7ff98003bce0 2021-03-09T09:53:04.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.176+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800112c0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff994081700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:04.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081340 con 0x7ff98003bce0 2021-03-09T09:53:04.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940af8f0 con 0x7ff98003bce0 2021-03-09T09:53:04.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.180+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af8f0 con 0x7ff98003bce0 2021-03-09T09:53:04.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.180+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:04.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.180+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:04.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:04.180+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:09.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800112c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:09.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800112c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:09.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800112c0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff994081700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:09.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800112c0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff994081700).stop 2021-03-09T09:53:09.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff9800112c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:09.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff9800112c0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:09.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:09.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:09.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:09.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097340 con 0x7ff99c0adc20 2021-03-09T09:53:09.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014d60 con 0x7ff99c0613a0 2021-03-09T09:53:09.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001afe0 con 0x7ff980051540 2021-03-09T09:53:09.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:09.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:09.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:09.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:09.182 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:09.171+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5915 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:09.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:09.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:09.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:09.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:09.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:09.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008e390 con 0x7ff980051540 2021-03-09T09:53:09.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.171+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800416c0 con 0x7ff980051540 2021-03-09T09:53:09.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.175+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012310 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 rx=0x7ff980019ba0 tx=0x7ff994081b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:09.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069030 con 0x7ff980051540 2021-03-09T09:53:09.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994027560 con 0x7ff980051540 2021-03-09T09:53:09.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994005a40 con 0x7ff980051540 2021-03-09T09:53:09.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.179+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:09.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.179+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:09.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:09.179+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:14.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:53:14.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:14.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012310 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 rx=0x7ff980019ba0 tx=0x7ff994081b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:14.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012310 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 rx=0x7ff980019ba0 tx=0x7ff994081b20).stop 2021-03-09T09:53:14.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:14.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012310 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:14.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:14.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:14.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:14.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a400 con 0x7ff99c0adc20 2021-03-09T09:53:14.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051bc0 con 0x7ff99c0613a0 2021-03-09T09:53:14.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008ad0 con 0x7ff98003bce0 2021-03-09T09:53:14.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:14.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:14.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:14.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:14.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:14.184 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:14.175+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5955 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:14.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:14.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:14.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:14.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:14.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002f460 con 0x7ff98003bce0 2021-03-09T09:53:14.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.175+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff98003bce0 2021-03-09T09:53:14.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.179+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012800 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7ff98c005d80 tx=0x7ff98c006af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:14.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff98003bce0 2021-03-09T09:53:14.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff98003bce0 2021-03-09T09:53:14.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98003bce0 2021-03-09T09:53:14.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.183+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:14.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.183+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:14.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:14.183+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:19.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980012800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:19.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980012800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:19.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012800 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7ff98c005d80 tx=0x7ff98c006af0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:19.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012800 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7ff98c005d80 tx=0x7ff98c006af0).stop 2021-03-09T09:53:19.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980012800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:19.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012800 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:19.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980011120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:19.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:19.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:19.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012e90 con 0x7ff99c0adc20 2021-03-09T09:53:19.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f4b0 con 0x7ff99c0613a0 2021-03-09T09:53:19.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013cf0 con 0x7ff980051540 2021-03-09T09:53:19.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:19.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:19.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980011120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:19.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:19.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:19.186 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:19.175+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5985 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:19.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:19.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:19.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:19.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:19.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047d00 con 0x7ff980051540 2021-03-09T09:53:19.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.175+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980032ac0 con 0x7ff980051540 2021-03-09T09:53:19.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.179+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980011120 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984006c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:19.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840270d0 con 0x7ff980051540 2021-03-09T09:53:19.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98407f790 con 0x7ff980051540 2021-03-09T09:53:19.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98407f790 con 0x7ff980051540 2021-03-09T09:53:19.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.183+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:19.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.183+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:19.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:19.183+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:24.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980011120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:24.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980011120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:24.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980011120 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984006c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:24.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980011120 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984006c60).stop 2021-03-09T09:53:24.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980011120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:24.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980011120 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:24.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:24.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:24.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98001b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:24.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097340 con 0x7ff99c0adc20 2021-03-09T09:53:24.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002ce80 con 0x7ff99c0613a0 2021-03-09T09:53:24.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:24.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052aa0 con 0x7ff98003bce0 2021-03-09T09:53:24.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:24.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:24.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:24.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98001b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:24.188 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:24.179+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6028 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:24.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:24.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:24.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:24.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:24.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800432f0 con 0x7ff98003bce0 2021-03-09T09:53:24.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008e390 con 0x7ff98003bce0 2021-03-09T09:53:24.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98001b390 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7ff99400b0b0 tx=0x7ff9940070c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:24.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff98003bce0 2021-03-09T09:53:24.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994082e80 con 0x7ff98003bce0 2021-03-09T09:53:24.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98003bce0 2021-03-09T09:53:24.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.191+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:24.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.191+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:24.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:24.191+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:26.415 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:53:26.418 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:53:29.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98001b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:29.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98001b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:29.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98001b390 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7ff99400b0b0 tx=0x7ff9940070c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:29.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98001b390 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7ff99400b0b0 tx=0x7ff9940070c0).stop 2021-03-09T09:53:29.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff98001b390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:29.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff98001b390 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:29.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:29.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:29.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:29.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017910 con 0x7ff99c0adc20 2021-03-09T09:53:29.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002ce80 con 0x7ff99c0613a0 2021-03-09T09:53:29.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097340 con 0x7ff980051540 2021-03-09T09:53:29.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:29.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:29.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:29.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:29.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.179+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:29.190 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:29.179+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6064 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:29.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:29.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:29.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:29.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:29.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:53:29.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002f460 con 0x7ff980051540 2021-03-09T09:53:29.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b880 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 rx=0x7ff98c045ff0 tx=0x7ff98c046110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:29.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053030 con 0x7ff980051540 2021-03-09T09:53:29.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980051540 2021-03-09T09:53:29.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T09:53:29.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.191+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:29.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.191+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:29.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:29.191+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:34.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98001b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:34.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98001b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:34.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b880 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 rx=0x7ff98c045ff0 tx=0x7ff98c046110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:34.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b880 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 rx=0x7ff98c045ff0 tx=0x7ff98c046110).stop 2021-03-09T09:53:34.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98001b880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:34.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b880 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:34.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:34.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:34.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:34.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017910 con 0x7ff99c0adc20 2021-03-09T09:53:34.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097340 con 0x7ff99c0613a0 2021-03-09T09:53:34.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022130 con 0x7ff98003bce0 2021-03-09T09:53:34.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:34.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.179+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:34.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:34.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:34.192 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:34.183+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6100 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:34.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.183+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:34.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.183+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:34.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.183+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:34.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.183+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:34.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.183+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:34.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.183+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003ab00 con 0x7ff98003bce0 2021-03-09T09:53:34.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.183+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047d00 con 0x7ff98003bce0 2021-03-09T09:53:34.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.191+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 rx=0x7ff9840047a0 tx=0x7ff984006aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:34.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98400f260 con 0x7ff98003bce0 2021-03-09T09:53:34.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400f260 con 0x7ff98003bce0 2021-03-09T09:53:34.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff98003bce0 2021-03-09T09:53:34.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.195+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:34.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.195+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:34.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:34.195+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:39.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:39.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:39.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 rx=0x7ff9840047a0 tx=0x7ff984006aa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:39.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 rx=0x7ff9840047a0 tx=0x7ff984006aa0).stop 2021-03-09T09:53:39.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980011120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:39.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980011120 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:39.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:39.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:39.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:39.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017910 con 0x7ff99c0adc20 2021-03-09T09:53:39.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f4b0 con 0x7ff99c0613a0 2021-03-09T09:53:39.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050740 con 0x7ff980051540 2021-03-09T09:53:39.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:39.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:39.193 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:39.183+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6136 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:39.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:39.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:39.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:39.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:39.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:39.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:39.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:39.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004ac60 con 0x7ff980051540 2021-03-09T09:53:39.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.183+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800432f0 con 0x7ff980051540 2021-03-09T09:53:39.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.187+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b390 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940814f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:39.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980051540 2021-03-09T09:53:39.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980051540 2021-03-09T09:53:39.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:53:39.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.195+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:39.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.195+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:39.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:39.195+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:44.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98001b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:44.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98001b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:44.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b390 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940814f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:44.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b390 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940814f0).stop 2021-03-09T09:53:44.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98001b390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:44.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98001b390 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:44.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:44.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:44.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:44.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097340 con 0x7ff99c0adc20 2021-03-09T09:53:44.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f4b0 con 0x7ff99c0613a0 2021-03-09T09:53:44.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017910 con 0x7ff98003bce0 2021-03-09T09:53:44.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:44.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:44.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:44.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:44.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:44.195 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:44.187+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6170 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:44.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:44.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:44.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:44.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:44.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980042890 con 0x7ff98003bce0 2021-03-09T09:53:44.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff98003bce0 2021-03-09T09:53:44.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012220 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 rx=0x7ff98c084e50 tx=0x7ff98c079250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:44.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053080 con 0x7ff98003bce0 2021-03-09T09:53:44.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff98003bce0 2021-03-09T09:53:44.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98003bce0 2021-03-09T09:53:44.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.195+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:44.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.195+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:44.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:44.195+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:49.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980012220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:49.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980012220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:49.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012220 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 rx=0x7ff98c084e50 tx=0x7ff98c079250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:49.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012220 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 rx=0x7ff98c084e50 tx=0x7ff98c079250).stop 2021-03-09T09:53:49.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980012220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:49.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980012220 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:49.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:49.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:49.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:49.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d300 con 0x7ff99c0adc20 2021-03-09T09:53:49.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:49.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097340 con 0x7ff99c0613a0 2021-03-09T09:53:49.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038320 con 0x7ff980051540 2021-03-09T09:53:49.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:49.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:49.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:49.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:49.197 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:49.187+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6206 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:49.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.191+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:49.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.191+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:49.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.191+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:49.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.191+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:49.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.191+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980037430 con 0x7ff980051540 2021-03-09T09:53:49.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.191+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980042890 con 0x7ff980051540 2021-03-09T09:53:49.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.191+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 rx=0x7ff98c045c50 tx=0x7ff98c045cc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:49.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980051540 2021-03-09T09:53:49.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04d040 con 0x7ff980051540 2021-03-09T09:53:49.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079b60 con 0x7ff980051540 2021-03-09T09:53:49.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.199+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:49.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.199+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:49.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.199+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.891+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 msgr2=0x7ff9800855c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000137) for more than 900000000 us, fault. 2021-03-09T09:53:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800855c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7ff9940092d0 tx=0x7ff994017000).stop 2021-03-09T09:53:49.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.891+0000 7ff9937fe700 4 mgrc ms_handle_reset ms_handle_reset con 0x7ff980083100 2021-03-09T09:53:49.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.891+0000 7ff9937fe700 4 mgrc reconnect Terminating session with v2:172.21.15.52:6808/14433 2021-03-09T09:53:49.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 msgr2=0x7ff9800855c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:49.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.891+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800855c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.891+0000 7ff9937fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.891+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff98003bce0 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff98003bce0 0x7ff980021c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:49.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:49.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff98003bce0 0x7ff980021c10 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7ff99407c040 tx=0x7ff994017000).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:54.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:54.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:54.196 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:54.187+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6243 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:54.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 rx=0x7ff98c045c50 tx=0x7ff98c045cc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:54.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 rx=0x7ff98c045c50 tx=0x7ff98c045cc0).stop 2021-03-09T09:53:54.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:54.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:54.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:54.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:54.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:54.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d300 con 0x7ff99c0adc20 2021-03-09T09:53:54.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980085a30 con 0x7ff99c0613a0 2021-03-09T09:53:54.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097340 con 0x7ff980083100 2021-03-09T09:53:54.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:54.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:54.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.191+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:54.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.191+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:54.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.191+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:54.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:54.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.195+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:54.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:54.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.195+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:54.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980008140 con 0x7ff980083100 2021-03-09T09:53:54.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003ab00 con 0x7ff980083100 2021-03-09T09:53:54.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.195+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085540 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff9840794a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:54.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840270d0 con 0x7ff980083100 2021-03-09T09:53:54.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840270d0 con 0x7ff980083100 2021-03-09T09:53:54.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984033610 con 0x7ff980083100 2021-03-09T09:53:54.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.199+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:54.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.199+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:54.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:54.199+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:53:56.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:53:56.600 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:53:59.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980085540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:59.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980085540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:59.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085540 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff9840794a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:59.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085540 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff9840794a0).stop 2021-03-09T09:53:59.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980085540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:59.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085540 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:59.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:59.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:59.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:59.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980085a30 con 0x7ff99c0adc20 2021-03-09T09:53:59.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800118f0 con 0x7ff99c0613a0 2021-03-09T09:53:59.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:53:59.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800364a0 con 0x7ff980051540 2021-03-09T09:53:59.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:59.201 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:53:59.191+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6275 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:53:59.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:53:59.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:59.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:59.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:59.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:59.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:59.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:59.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.191+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980032ac0 con 0x7ff980051540 2021-03-09T09:53:59.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004ac60 con 0x7ff980051540 2021-03-09T09:53:59.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7ff994036010 tx=0x7ff9940814f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:59.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T09:53:59.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980051540 2021-03-09T09:53:59.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:53:59.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.203+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:59.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.203+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:59.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:53:59.203+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:04.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:04.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:04.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7ff994036010 tx=0x7ff9940814f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:04.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7ff994036010 tx=0x7ff9940814f0).stop 2021-03-09T09:54:04.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:04.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012710 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:04.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:04.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:04.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:04.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097340 con 0x7ff99c0adc20 2021-03-09T09:54:04.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d300 con 0x7ff99c0613a0 2021-03-09T09:54:04.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004d60 con 0x7ff980083100 2021-03-09T09:54:04.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:04.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:04.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:04.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:04.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.191+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:04.203 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:04.191+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6311 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:04.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:04.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.195+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:04.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:04.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.195+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:04.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980052aa0 con 0x7ff980083100 2021-03-09T09:54:04.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980008140 con 0x7ff980083100 2021-03-09T09:54:04.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.195+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085520 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff98403f250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:04.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019040 con 0x7ff980083100 2021-03-09T09:54:04.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984027080 con 0x7ff980083100 2021-03-09T09:54:04.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984033610 con 0x7ff980083100 2021-03-09T09:54:04.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.203+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:04.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.207+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:04.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:04.207+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:09.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980085520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:09.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980085520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:09.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085520 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff98403f250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:09.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085520 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff98403f250).stop 2021-03-09T09:54:09.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980085520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:09.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980085520 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:09.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:09.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:09.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:09.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030570 con 0x7ff99c0adc20 2021-03-09T09:54:09.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004bdb0 con 0x7ff99c0613a0 2021-03-09T09:54:09.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c9a0 con 0x7ff980051540 2021-03-09T09:54:09.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:09.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:09.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:09.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:09.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:09.205 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:09.195+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6350 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:09.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:09.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:09.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:09.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:09.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:54:09.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980032ac0 con 0x7ff980051540 2021-03-09T09:54:09.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012a90 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7ff994006fa0 tx=0x7ff99408c910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:09.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994040070 con 0x7ff980051540 2021-03-09T09:54:09.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994074e90 con 0x7ff980051540 2021-03-09T09:54:09.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:54:09.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.203+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:09.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.207+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:09.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:09.207+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:14.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:14.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:14.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012a90 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7ff994006fa0 tx=0x7ff99408c910).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:14.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012a90 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7ff994006fa0 tx=0x7ff99408c910).stop 2021-03-09T09:54:14.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980012a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:14.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980012a90 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:14.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:14.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:14.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:14.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800117b0 con 0x7ff99c0adc20 2021-03-09T09:54:14.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d300 con 0x7ff99c0613a0 2021-03-09T09:54:14.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008ad0 con 0x7ff980083100 2021-03-09T09:54:14.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:14.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:14.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:14.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:14.206 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:14.195+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6388 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:14.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.195+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:14.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:14.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:14.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:14.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:14.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008da60 con 0x7ff980083100 2021-03-09T09:54:14.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980037430 con 0x7ff980083100 2021-03-09T09:54:14.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 rx=0x7ff98c0a1010 tx=0x7ff98c079340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:14.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.207+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980083100 2021-03-09T09:54:14.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.207+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00ba80 con 0x7ff980083100 2021-03-09T09:54:14.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.207+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079b60 con 0x7ff980083100 2021-03-09T09:54:14.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.207+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:14.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.207+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:14.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:14.207+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:19.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:19.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:19.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 rx=0x7ff98c0a1010 tx=0x7ff98c079340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:19.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 rx=0x7ff98c0a1010 tx=0x7ff98c079340).stop 2021-03-09T09:54:19.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:19.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:19.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:19.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:19.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:19.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b000 con 0x7ff99c0adc20 2021-03-09T09:54:19.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005940 con 0x7ff99c0613a0 2021-03-09T09:54:19.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e240 con 0x7ff980051540 2021-03-09T09:54:19.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:19.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:19.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:19.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:19.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:19.210 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:19.199+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6419 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:19.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:19.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:19.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:19.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:19.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980031220 con 0x7ff980051540 2021-03-09T09:54:19.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008da60 con 0x7ff980051540 2021-03-09T09:54:19.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c008180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:19.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075050 con 0x7ff980051540 2021-03-09T09:54:19.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d090 con 0x7ff980051540 2021-03-09T09:54:19.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.207+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980051540 2021-03-09T09:54:19.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.207+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:19.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.207+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:19.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:19.207+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:24.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:24.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:24.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c008180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:24.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c008180).stop 2021-03-09T09:54:24.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:24.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:24.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:24.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:24.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:24.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005940 con 0x7ff99c0adc20 2021-03-09T09:54:24.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001c880 con 0x7ff99c0613a0 2021-03-09T09:54:24.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e3e0 con 0x7ff980083100 2021-03-09T09:54:24.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:24.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:24.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:24.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:24.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:24.210 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:24.199+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6454 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:24.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:24.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:24.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:24.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:24.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800401b0 con 0x7ff980083100 2021-03-09T09:54:24.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.199+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980052aa0 con 0x7ff980083100 2021-03-09T09:54:24.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.203+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff984081450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:24.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840270e0 con 0x7ff980083100 2021-03-09T09:54:24.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400db90 con 0x7ff980083100 2021-03-09T09:54:24.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.207+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98402c180 con 0x7ff980083100 2021-03-09T09:54:24.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.207+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:24.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.211+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:24.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:24.211+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:26.636 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:54:26.639 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:54:29.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:29.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:29.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff984081450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:29.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff984081450).stop 2021-03-09T09:54:29.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:29.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:29.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:29.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:29.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:29.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005940 con 0x7ff99c0adc20 2021-03-09T09:54:29.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d060 con 0x7ff99c0613a0 2021-03-09T09:54:29.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013090 con 0x7ff980051540 2021-03-09T09:54:29.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:29.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:29.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:29.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:29.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.202+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:29.212 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:29.202+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6496 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:29.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:29.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:29.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:29.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:29.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800193a0 con 0x7ff980051540 2021-03-09T09:54:29.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980031220 con 0x7ff980051540 2021-03-09T09:54:29.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c07def0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:29.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T09:54:29.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980051540 2021-03-09T09:54:29.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T09:54:29.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.218+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:29.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.218+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:29.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:29.218+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:34.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:54:34.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:34.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c07def0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:34.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c07def0).stop 2021-03-09T09:54:34.211 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:34.202+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6530 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:34.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:34.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037fe0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:34.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:34.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:34.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:34.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011ec0 con 0x7ff99c0adc20 2021-03-09T09:54:34.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d6d0 con 0x7ff99c0613a0 2021-03-09T09:54:34.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013090 con 0x7ff980083100 2021-03-09T09:54:34.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:34.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:34.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:34.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.202+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:34.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.206+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:34.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.206+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:34.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.206+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:34.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.206+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:34.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.206+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:34.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.206+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004a520 con 0x7ff980083100 2021-03-09T09:54:34.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.206+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800193a0 con 0x7ff980083100 2021-03-09T09:54:34.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.210+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7ff98c07cdf0 tx=0x7ff98c008180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:34.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff980083100 2021-03-09T09:54:34.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08d900 con 0x7ff980083100 2021-03-09T09:54:34.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T09:54:34.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.214+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:34.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.214+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:34.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:34.214+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:39.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:39.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:39.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7ff98c07cdf0 tx=0x7ff98c008180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:39.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7ff98c07cdf0 tx=0x7ff98c008180).stop 2021-03-09T09:54:39.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:39.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012020 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:39.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:39.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:39.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980031080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:39.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013090 con 0x7ff99c0adc20 2021-03-09T09:54:39.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:39.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:39.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036480 con 0x7ff99c0613a0 2021-03-09T09:54:39.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:39.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:39.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce80 con 0x7ff980051540 2021-03-09T09:54:39.216 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:39.206+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6570 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:39.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980031080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:39.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:39.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:39.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:39.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:39.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004ac60 con 0x7ff980051540 2021-03-09T09:54:39.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:54:39.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.210+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980031080 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 rx=0x7ff980013df0 tx=0x7ff99403edb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:39.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994031dd0 con 0x7ff980051540 2021-03-09T09:54:39.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994031dd0 con 0x7ff980051540 2021-03-09T09:54:39.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:54:39.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.218+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:39.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.218+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:39.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:39.218+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:44.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980031080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:44.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980031080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:44.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980031080 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 rx=0x7ff980013df0 tx=0x7ff99403edb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:44.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980031080 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 rx=0x7ff980013df0 tx=0x7ff99403edb0).stop 2021-03-09T09:54:44.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980031080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:44.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980031080 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:44.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:44.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:44.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:44.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980040be0 con 0x7ff99c0adc20 2021-03-09T09:54:44.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013090 con 0x7ff99c0613a0 2021-03-09T09:54:44.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800572c0 con 0x7ff980083100 2021-03-09T09:54:44.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:44.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:44.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:44.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:44.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:44.218 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:44.206+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6603 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:44.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:44.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:44.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:44.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:44.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980043630 con 0x7ff980083100 2021-03-09T09:54:44.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004ac60 con 0x7ff980083100 2021-03-09T09:54:44.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011db0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994081b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:44.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980083100 2021-03-09T09:54:44.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009fb0 con 0x7ff980083100 2021-03-09T09:54:44.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99400a120 con 0x7ff980083100 2021-03-09T09:54:44.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.210+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:44.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.210+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:44.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:44.210+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:49.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980011db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:49.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980011db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:49.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011db0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994081b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:49.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011db0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994081b20).stop 2021-03-09T09:54:49.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980011db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:49.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011db0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:49.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:49.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:49.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:49.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001afe0 con 0x7ff99c0adc20 2021-03-09T09:54:49.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800448c0 con 0x7ff99c0613a0 2021-03-09T09:54:49.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036480 con 0x7ff980051540 2021-03-09T09:54:49.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:49.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:49.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:49.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:49.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.206+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:49.219 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:49.206+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6640 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:49.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.210+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:49.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.210+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:49.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.210+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:49.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.210+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:49.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.210+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:54:49.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.210+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800401b0 con 0x7ff980051540 2021-03-09T09:54:49.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.210+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fce0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7ff984006ce0 tx=0x7ff984035420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:49.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984027120 con 0x7ff980051540 2021-03-09T09:54:49.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984009030 con 0x7ff980051540 2021-03-09T09:54:49.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98402c910 con 0x7ff980051540 2021-03-09T09:54:49.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.218+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:49.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.218+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:49.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:49.218+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:54.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:54.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:54.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fce0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7ff984006ce0 tx=0x7ff984035420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:54.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fce0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7ff984006ce0 tx=0x7ff984035420).stop 2021-03-09T09:54:54.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002fce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:54.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fce0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:54.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:54.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:54.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:54.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050420 con 0x7ff99c0adc20 2021-03-09T09:54:54.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800572c0 con 0x7ff99c0613a0 2021-03-09T09:54:54.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:54.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008ad0 con 0x7ff980083100 2021-03-09T09:54:54.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:54.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:54.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:54.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:54.223 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:54.210+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6676 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:54.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:54.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:54.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:54.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:54.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055520 con 0x7ff980083100 2021-03-09T09:54:54.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.210+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004a520 con 0x7ff980083100 2021-03-09T09:54:54.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011d60 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c00baf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:54.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075090 con 0x7ff980083100 2021-03-09T09:54:54.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d080 con 0x7ff980083100 2021-03-09T09:54:54.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980083100 2021-03-09T09:54:54.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.222+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:54.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.222+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:54.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:54.222+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:54:56.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:54:56.682 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:54:59.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980011d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:59.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980011d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:59.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011d60 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c00baf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:59.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011d60 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c00baf0).stop 2021-03-09T09:54:59.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980011d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:59.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980011d60 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:59.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:59.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:59.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:59.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012590 con 0x7ff99c0adc20 2021-03-09T09:54:59.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014120 con 0x7ff99c0613a0 2021-03-09T09:54:59.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c4a0 con 0x7ff980051540 2021-03-09T09:54:59.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:54:59.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:54:59.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:59.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:59.223 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:54:59.214+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6708 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:54:59.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:59.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:59.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:59.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:59.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:59.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008da60 con 0x7ff980051540 2021-03-09T09:54:59.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055520 con 0x7ff980051540 2021-03-09T09:54:59.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fc50 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c0a2f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:59.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980051540 2021-03-09T09:54:59.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980051540 2021-03-09T09:54:59.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff980051540 2021-03-09T09:54:59.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.222+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:59.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.222+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:59.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:54:59.222+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:04.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:04.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:04.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fc50 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c0a2f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:04.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fc50 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7ff98c07b1d0 tx=0x7ff98c0a2f10).stop 2021-03-09T09:55:04.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002fc50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:04.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002fc50 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:04.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:04.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:04.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:04.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020ff0 con 0x7ff99c0adc20 2021-03-09T09:55:04.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014700 con 0x7ff99c0613a0 2021-03-09T09:55:04.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:04.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980042b10 con 0x7ff980083100 2021-03-09T09:55:04.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:04.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:04.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:04.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:04.225 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:04.214+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6748 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:04.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:04.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:04.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:04.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:04.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800193a0 con 0x7ff980083100 2021-03-09T09:55:04.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980043630 con 0x7ff980083100 2021-03-09T09:55:04.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037fe0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99403ee20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:04.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994040070 con 0x7ff980083100 2021-03-09T09:55:04.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T09:55:04.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T09:55:04.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.222+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:04.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.222+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:04.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:04.222+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:09.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:55:09.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:09.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037fe0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99403ee20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:09.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037fe0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99403ee20).stop 2021-03-09T09:55:09.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:09.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037fe0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:09.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:09.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800300f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:09.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:09.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013ab0 con 0x7ff99c0adc20 2021-03-09T09:55:09.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f880 con 0x7ff99c0613a0 2021-03-09T09:55:09.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800148a0 con 0x7ff980051540 2021-03-09T09:55:09.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:09.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:09.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:09.226 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:09.214+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6781 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:09.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:09.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800300f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:09.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.218+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:09.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.218+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:09.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.218+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:09.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.218+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:09.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.218+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004ac60 con 0x7ff980051540 2021-03-09T09:55:09.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.218+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008da60 con 0x7ff980051540 2021-03-09T09:55:09.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.218+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800300f0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7ff980031570 tx=0x7ff98c022fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:09.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T09:55:09.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a0130 con 0x7ff980051540 2021-03-09T09:55:09.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T09:55:09.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.230+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:09.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.230+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:09.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:09.230+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:14.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800300f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:14.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800300f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:14.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800300f0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7ff980031570 tx=0x7ff98c022fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:14.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800300f0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7ff980031570 tx=0x7ff98c022fe0).stop 2021-03-09T09:55:14.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800300f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:14.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800300f0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:14.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:14.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:14.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:14.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013ab0 con 0x7ff99c0adc20 2021-03-09T09:55:14.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097560 con 0x7ff99c0613a0 2021-03-09T09:55:14.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a230 con 0x7ff980083100 2021-03-09T09:55:14.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:14.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:14.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:14.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:14.228 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:14.218+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6820 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:14.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:14.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:14.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:14.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:14.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:14.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980032ac0 con 0x7ff980083100 2021-03-09T09:55:14.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.218+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800193a0 con 0x7ff980083100 2021-03-09T09:55:14.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.222+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7ff9800110a0 tx=0x7ff99400a4e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:14.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980083100 2021-03-09T09:55:14.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a250 con 0x7ff980083100 2021-03-09T09:55:14.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T09:55:14.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.226+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:14.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.226+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:14.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:14.226+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:19.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:19.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:19.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7ff9800110a0 tx=0x7ff99400a4e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:19.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7ff9800110a0 tx=0x7ff99400a4e0).stop 2021-03-09T09:55:19.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:19.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:19.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:19.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:19.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:19.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004fac0 con 0x7ff99c0adc20 2021-03-09T09:55:19.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d920 con 0x7ff99c0613a0 2021-03-09T09:55:19.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097560 con 0x7ff980051540 2021-03-09T09:55:19.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:19.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:19.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:19.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:19.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:19.233 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:19.222+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6857 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:19.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:19.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:19.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:19.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:19.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800264f0 con 0x7ff980051540 2021-03-09T09:55:19.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:55:19.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093e40 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7ff98407b790 tx=0x7ff98400f2f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:19.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980051540 2021-03-09T09:55:19.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984044930 con 0x7ff980051540 2021-03-09T09:55:19.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980051540 2021-03-09T09:55:19.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.226+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:19.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.230+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:19.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:19.230+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:24.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:24.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:24.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093e40 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7ff98407b790 tx=0x7ff98400f2f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:24.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093e40 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7ff98407b790 tx=0x7ff98400f2f0).stop 2021-03-09T09:55:24.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:24.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093e40 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:24.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:24.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980008500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:24.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:24.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004fac0 con 0x7ff99c0adc20 2021-03-09T09:55:24.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000bbe0 con 0x7ff99c0613a0 2021-03-09T09:55:24.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097560 con 0x7ff980083100 2021-03-09T09:55:24.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:24.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:24.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:24.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:24.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980008500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:24.233 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:24.222+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6892 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:24.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:24.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:24.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:24.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:24.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980042890 con 0x7ff980083100 2021-03-09T09:55:24.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004ac60 con 0x7ff980083100 2021-03-09T09:55:24.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980008500 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7ff980011730 tx=0x7ff98c004110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:24.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff980083100 2021-03-09T09:55:24.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a0130 con 0x7ff980083100 2021-03-09T09:55:24.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff980083100 2021-03-09T09:55:24.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.230+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:24.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.230+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:24.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:24.230+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:26.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:55:26.789 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:55:29.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980008500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:29.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980008500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:29.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980008500 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7ff980011730 tx=0x7ff98c004110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:29.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980008500 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7ff980011730 tx=0x7ff98c004110).stop 2021-03-09T09:55:29.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980008500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:29.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980008500 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:29.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:29.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:29.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980019960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:29.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800106c0 con 0x7ff99c0adc20 2021-03-09T09:55:29.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037900 con 0x7ff99c0613a0 2021-03-09T09:55:29.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980042b10 con 0x7ff980051540 2021-03-09T09:55:29.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:29.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:29.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:29.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:29.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980019960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:29.235 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:29.226+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6931 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:29.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:29.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:29.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:29.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:29.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:55:29.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980032ac0 con 0x7ff980051540 2021-03-09T09:55:29.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980019960 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994066310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:29.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980051540 2021-03-09T09:55:29.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081bb0 con 0x7ff980051540 2021-03-09T09:55:29.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.234+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:55:29.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.234+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:29.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.234+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:29.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:29.234+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:34.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980019960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:34.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980019960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:34.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980019960 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994066310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:34.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980019960 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994066310).stop 2021-03-09T09:55:34.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980019960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:34.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980019960 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:34.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:34.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:34.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:34.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056a30 con 0x7ff99c0adc20 2021-03-09T09:55:34.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029a60 con 0x7ff99c0613a0 2021-03-09T09:55:34.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013ab0 con 0x7ff980083100 2021-03-09T09:55:34.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:34.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:34.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:34.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:34.237 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:34.226+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6964 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:34.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:34.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:34.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:34.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:34.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:34.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d360 con 0x7ff980083100 2021-03-09T09:55:34.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.226+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800264f0 con 0x7ff980083100 2021-03-09T09:55:34.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.230+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7ff9840328a0 tx=0x7ff984088980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:34.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840355d0 con 0x7ff980083100 2021-03-09T09:55:34.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.234+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840355d0 con 0x7ff980083100 2021-03-09T09:55:34.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.238+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980083100 2021-03-09T09:55:34.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.238+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:34.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.238+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:34.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:34.238+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:39.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:39.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:39.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7ff9840328a0 tx=0x7ff984088980).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:39.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7ff9840328a0 tx=0x7ff984088980).stop 2021-03-09T09:55:39.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:39.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038710 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:39.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:39.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:39.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800887e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:39.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b7b0 con 0x7ff99c0adc20 2021-03-09T09:55:39.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800319b0 con 0x7ff99c0613a0 2021-03-09T09:55:39.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980042b10 con 0x7ff980051540 2021-03-09T09:55:39.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:39.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:39.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:39.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:39.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800887e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:39.244 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:39.230+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6997 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:39.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:39.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:39.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:39.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:39.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008da60 con 0x7ff980051540 2021-03-09T09:55:39.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050d30 con 0x7ff980051540 2021-03-09T09:55:39.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800887e0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994031300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:39.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.234+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980051540 2021-03-09T09:55:39.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.238+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940813a0 con 0x7ff980051540 2021-03-09T09:55:39.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.238+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:55:39.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.238+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:39.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.242+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:39.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:39.242+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:44.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800887e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:44.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800887e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:44.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800887e0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994031300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:44.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800887e0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994031300).stop 2021-03-09T09:55:44.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800887e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:44.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800887e0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:44.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:44.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:44.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:44.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056a30 con 0x7ff99c0adc20 2021-03-09T09:55:44.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029a60 con 0x7ff99c0613a0 2021-03-09T09:55:44.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013ab0 con 0x7ff980083100 2021-03-09T09:55:44.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:44.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:44.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:44.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:44.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:44.244 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:44.230+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7036 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:44.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:44.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:44.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:44.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:44.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800193a0 con 0x7ff980083100 2021-03-09T09:55:44.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980042890 con 0x7ff980083100 2021-03-09T09:55:44.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.230+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088cd0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7ff9800936b0 tx=0x7ff98c03d1b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:44.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.234+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c07b090 con 0x7ff980083100 2021-03-09T09:55:44.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.238+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d090 con 0x7ff980083100 2021-03-09T09:55:44.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.238+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff980083100 2021-03-09T09:55:44.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.238+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:44.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.242+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:44.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:44.242+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:49.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980088cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:49.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980088cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:49.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088cd0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7ff9800936b0 tx=0x7ff98c03d1b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:49.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088cd0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7ff9800936b0 tx=0x7ff98c03d1b0).stop 2021-03-09T09:55:49.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980088cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:49.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088cd0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:49.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:49.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:49.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:49.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800896e0 con 0x7ff99c0adc20 2021-03-09T09:55:49.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043b30 con 0x7ff99c0613a0 2021-03-09T09:55:49.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980042b10 con 0x7ff980051540 2021-03-09T09:55:49.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:49.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:49.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:49.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:49.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:49.242 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:49.233+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7073 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:49.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:49.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:49.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:49.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:49.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004ac60 con 0x7ff980051540 2021-03-09T09:55:49.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008da60 con 0x7ff980051540 2021-03-09T09:55:49.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff994066000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:49.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T09:55:49.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407be50 con 0x7ff980051540 2021-03-09T09:55:49.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.237+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:55:49.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.237+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:49.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.237+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:49.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:49.237+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:54.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800394b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:54.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800394b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:54.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff994066000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:54.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff994066000).stop 2021-03-09T09:55:54.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800394b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:54.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:54.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:54.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:54.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:54.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980042b10 con 0x7ff99c0adc20 2021-03-09T09:55:54.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016a60 con 0x7ff99c0613a0 2021-03-09T09:55:54.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036480 con 0x7ff980083100 2021-03-09T09:55:54.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:54.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:54.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:54.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:54.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:54.245 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:54.233+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7109 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:54.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:54.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:54.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:54.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:54.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000c460 con 0x7ff980083100 2021-03-09T09:55:54.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.233+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004ac60 con 0x7ff980083100 2021-03-09T09:55:54.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.237+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019700 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7ff98008d120 tx=0x7ff994097930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:54.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.237+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980083100 2021-03-09T09:55:54.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.237+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940abed0 con 0x7ff980083100 2021-03-09T09:55:54.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940abac0 con 0x7ff980083100 2021-03-09T09:55:54.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.241+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:54.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.241+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:54.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:54.241+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:55:56.831 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:55:56.834 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:55:59.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980019700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:55:59.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980019700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:59.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019700 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7ff98008d120 tx=0x7ff994097930).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:59.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019700 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7ff98008d120 tx=0x7ff994097930).stop 2021-03-09T09:55:59.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980019700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:59.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019700 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:59.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:59.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:59.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:59.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018520 con 0x7ff99c0adc20 2021-03-09T09:55:59.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036480 con 0x7ff99c0613a0 2021-03-09T09:55:59.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044580 con 0x7ff980051540 2021-03-09T09:55:59.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:55:59.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:59.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:55:59.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:59.247 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:55:59.237+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7140 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:55:59.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:59.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:59.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:59.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:59.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:59.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980043430 con 0x7ff980051540 2021-03-09T09:55:59.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d360 con 0x7ff980051540 2021-03-09T09:55:59.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.237+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff98400f1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:59.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840447f0 con 0x7ff980051540 2021-03-09T09:55:59.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984032000 con 0x7ff980051540 2021-03-09T09:55:59.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840447f0 con 0x7ff980051540 2021-03-09T09:55:59.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.245+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:59.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.245+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:59.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:55:59.245+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:04.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800394b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:04.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800394b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:04.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff98400f1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:04.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff98400f1f0).stop 2021-03-09T09:56:04.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800394b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:04.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800394b0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:04.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:04.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:04.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:04.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013170 con 0x7ff99c0adc20 2021-03-09T09:56:04.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036480 con 0x7ff99c0613a0 2021-03-09T09:56:04.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a0d0 con 0x7ff980083100 2021-03-09T09:56:04.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:04.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.233+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:04.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.237+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:04.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.237+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:04.248 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:04.237+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7171 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:04.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.237+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:04.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.237+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:04.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.237+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:04.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:04.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.241+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:04.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d360 con 0x7ff980083100 2021-03-09T09:56:04.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800193a0 con 0x7ff980083100 2021-03-09T09:56:04.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.241+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088810 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7ff98c008a90 tx=0x7ff98c079820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:04.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053020 con 0x7ff980083100 2021-03-09T09:56:04.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980083100 2021-03-09T09:56:04.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.249+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T09:56:04.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.249+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:04.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.249+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:04.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:04.249+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:09.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980088810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:09.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980088810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:09.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088810 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7ff98c008a90 tx=0x7ff98c079820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:09.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088810 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7ff98c008a90 tx=0x7ff98c079820).stop 2021-03-09T09:56:09.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980088810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:09.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088810 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:09.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:09.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:09.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:09.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013df0 con 0x7ff99c0adc20 2021-03-09T09:56:09.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800193f0 con 0x7ff99c0613a0 2021-03-09T09:56:09.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052550 con 0x7ff980051540 2021-03-09T09:56:09.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:09.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:09.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:09.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:09.250 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:09.241+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7205 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:09.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:09.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:09.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:09.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:09.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:09.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000da30 con 0x7ff980051540 2021-03-09T09:56:09.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d360 con 0x7ff980051540 2021-03-09T09:56:09.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.241+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088d00 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7ff98c07a280 tx=0x7ff98c04a990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:09.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053090 con 0x7ff980051540 2021-03-09T09:56:09.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff980051540 2021-03-09T09:56:09.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T09:56:09.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.253+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:09.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.253+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:09.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:09.253+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:14.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:14.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:14.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088d00 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7ff98c07a280 tx=0x7ff98c04a990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:14.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088d00 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7ff98c07a280 tx=0x7ff98c04a990).stop 2021-03-09T09:56:14.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:14.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088d00 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:14.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:14.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:14.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:14.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009450 con 0x7ff99c0adc20 2021-03-09T09:56:14.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044f40 con 0x7ff99c0613a0 2021-03-09T09:56:14.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980032250 con 0x7ff980083100 2021-03-09T09:56:14.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:14.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.241+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:14.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.245+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:14.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.245+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:14.252 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:14.245+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7249 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:14.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:14.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:14.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:14.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:14.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:14.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004ff80 con 0x7ff980083100 2021-03-09T09:56:14.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980043430 con 0x7ff980083100 2021-03-09T09:56:14.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ec0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7ff98400d510 tx=0x7ff9840813d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:14.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.249+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840a5970 con 0x7ff980083100 2021-03-09T09:56:14.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840a5970 con 0x7ff980083100 2021-03-09T09:56:14.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980083100 2021-03-09T09:56:14.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.257+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:14.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.257+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:14.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:14.257+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:19.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980018ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:19.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980018ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:19.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ec0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7ff98400d510 tx=0x7ff9840813d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:19.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ec0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7ff98400d510 tx=0x7ff9840813d0).stop 2021-03-09T09:56:19.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980018ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:19.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ec0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:19.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980089090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:19.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:19.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:19.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800139b0 con 0x7ff99c0adc20 2021-03-09T09:56:19.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009450 con 0x7ff99c0613a0 2021-03-09T09:56:19.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095730 con 0x7ff980051540 2021-03-09T09:56:19.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:19.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:19.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980089090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:19.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:19.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:19.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:19.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:19.255 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:19.245+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7285 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:19.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:19.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:19.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015470 con 0x7ff980051540 2021-03-09T09:56:19.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004ff80 con 0x7ff980051540 2021-03-09T09:56:19.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980089090 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984040bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:19.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980051540 2021-03-09T09:56:19.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840a5760 con 0x7ff980051540 2021-03-09T09:56:19.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980051540 2021-03-09T09:56:19.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.253+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:19.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.253+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:19.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:19.253+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:24.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980089090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:24.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980089090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:24.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980089090 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984040bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:24.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980089090 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984040bc0).stop 2021-03-09T09:56:24.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980089090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:24.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980089090 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:24.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:24.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800401b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:24.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:24.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009450 con 0x7ff99c0adc20 2021-03-09T09:56:24.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c520 con 0x7ff99c0613a0 2021-03-09T09:56:24.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003e3c0 con 0x7ff980083100 2021-03-09T09:56:24.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:24.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:24.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:24.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.245+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:24.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.249+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:24.256 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:24.249+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7323 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:24.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.249+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:24.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.249+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:24.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.249+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:24.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.249+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:24.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.249+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004ac60 con 0x7ff980083100 2021-03-09T09:56:24.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.249+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000da30 con 0x7ff980083100 2021-03-09T09:56:24.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.253+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800401b0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c004070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:24.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff980083100 2021-03-09T09:56:24.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980083100 2021-03-09T09:56:24.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980083100 2021-03-09T09:56:24.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.261+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:24.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.261+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:24.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:24.261+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:26.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:56:26.939 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:56:29.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:29.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:29.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800401b0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c004070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:29.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800401b0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c004070).stop 2021-03-09T09:56:29.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800401b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:29.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800401b0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:29.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:29.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:29.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:29.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800574e0 con 0x7ff99c0adc20 2021-03-09T09:56:29.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009450 con 0x7ff99c0613a0 2021-03-09T09:56:29.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093170 con 0x7ff980051540 2021-03-09T09:56:29.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:29.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:29.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:29.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:29.258 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:29.249+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7357 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:29.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.249+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:29.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:29.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.253+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:29.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:29.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.253+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:29.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d360 con 0x7ff980051540 2021-03-09T09:56:29.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000c460 con 0x7ff980051540 2021-03-09T09:56:29.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.253+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016950 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7ff98008c4a0 tx=0x7ff99403fcf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:29.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994099ea0 con 0x7ff980051540 2021-03-09T09:56:29.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:56:29.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980051540 2021-03-09T09:56:29.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.261+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:29.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.261+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:29.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:29.261+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:34.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:34.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:34.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016950 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7ff98008c4a0 tx=0x7ff99403fcf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:34.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016950 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7ff98008c4a0 tx=0x7ff99403fcf0).stop 2021-03-09T09:56:34.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:34.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016950 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:34.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:34.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:34.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:34.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009470 con 0x7ff99c0adc20 2021-03-09T09:56:34.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980032ac0 con 0x7ff99c0613a0 2021-03-09T09:56:34.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044720 con 0x7ff980083100 2021-03-09T09:56:34.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:34.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:34.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:34.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.249+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:34.260 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:34.253+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7390 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:34.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.253+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:34.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.253+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:34.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.253+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:34.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.253+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:34.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.253+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:34.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.253+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008b490 con 0x7ff980083100 2021-03-09T09:56:34.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.253+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004ac60 con 0x7ff980083100 2021-03-09T09:56:34.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.253+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008afa0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7ff98c03d000 tx=0x7ff98c07dcb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:34.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053080 con 0x7ff980083100 2021-03-09T09:56:34.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980083100 2021-03-09T09:56:34.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c03c020 con 0x7ff980083100 2021-03-09T09:56:34.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.261+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:34.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.265+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:34.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:34.265+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:39.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:56:39.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:39.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008afa0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7ff98c03d000 tx=0x7ff98c07dcb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:39.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008afa0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7ff98c03d000 tx=0x7ff98c07dcb0).stop 2021-03-09T09:56:39.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008afa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:39.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008afa0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:39.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:39.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:39.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:39.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800574e0 con 0x7ff99c0adc20 2021-03-09T09:56:39.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ae20 con 0x7ff99c0613a0 2021-03-09T09:56:39.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f840 con 0x7ff980051540 2021-03-09T09:56:39.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:39.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:39.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:39.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:39.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:39.262 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:39.253+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7426 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:39.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:39.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:39.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:39.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:39.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980044720 con 0x7ff980051540 2021-03-09T09:56:39.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d360 con 0x7ff980051540 2021-03-09T09:56:39.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.257+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003fcd0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7ff980097360 tx=0x7ff9940a9980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:39.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994040080 con 0x7ff980051540 2021-03-09T09:56:39.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:56:39.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:56:39.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.261+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:39.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.265+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:39.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:39.265+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:44.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:44.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:44.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003fcd0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7ff980097360 tx=0x7ff9940a9980).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:44.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003fcd0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7ff980097360 tx=0x7ff9940a9980).stop 2021-03-09T09:56:44.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003fcd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:44.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003fcd0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:44.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800188d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:44.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:44.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:44.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800574e0 con 0x7ff99c0adc20 2021-03-09T09:56:44.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800139b0 con 0x7ff99c0613a0 2021-03-09T09:56:44.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037930 con 0x7ff980083100 2021-03-09T09:56:44.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:44.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.253+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:44.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.257+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:44.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.257+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:44.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.257+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800188d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:44.264 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:44.257+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7468 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:44.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.257+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:44.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.257+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:44.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.257+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:44.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.257+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:44.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.257+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800150d0 con 0x7ff980083100 2021-03-09T09:56:44.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.257+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015470 con 0x7ff980083100 2021-03-09T09:56:44.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.257+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800188d0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff984040ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:44.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984006b70 con 0x7ff980083100 2021-03-09T09:56:44.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984027020 con 0x7ff980083100 2021-03-09T09:56:44.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.269+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840a5c50 con 0x7ff980083100 2021-03-09T09:56:44.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.269+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:44.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.269+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:44.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:44.269+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:49.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800188d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:56:49.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800188d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:49.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800188d0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff984040ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:49.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800188d0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff984040ca0).stop 2021-03-09T09:56:49.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800188d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:49.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800188d0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:49.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:49.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:49.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:49.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800574e0 con 0x7ff99c0adc20 2021-03-09T09:56:49.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093390 con 0x7ff99c0613a0 2021-03-09T09:56:49.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800139b0 con 0x7ff980051540 2021-03-09T09:56:49.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:49.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:49.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:49.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:49.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.257+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:49.267 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:49.257+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7501 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:49.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:49.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.261+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:49.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:49.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.261+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:49.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091ae0 con 0x7ff980051540 2021-03-09T09:56:49.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008b490 con 0x7ff980051540 2021-03-09T09:56:49.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.261+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7ff98c016280 tx=0x7ff98c005590).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:49.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.265+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980051540 2021-03-09T09:56:49.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.265+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980051540 2021-03-09T09:56:49.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.269+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T09:56:49.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.269+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:49.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.269+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:49.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:49.269+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:54.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980018dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:54.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980018dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:54.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7ff98c016280 tx=0x7ff98c005590).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:54.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7ff98c016280 tx=0x7ff98c005590).stop 2021-03-09T09:56:54.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980018dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:54.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:54.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:54.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:54.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:54.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800574e0 con 0x7ff99c0adc20 2021-03-09T09:56:54.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800139b0 con 0x7ff99c0613a0 2021-03-09T09:56:54.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.257+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b9b0 con 0x7ff980083100 2021-03-09T09:56:54.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:54.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:54.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:54.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:54.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:54.269 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:54.261+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7533 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:54.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:54.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:54.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:54.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:54.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800889b0 con 0x7ff980083100 2021-03-09T09:56:54.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091ae0 con 0x7ff980083100 2021-03-09T09:56:54.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.261+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ae20 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 rx=0x7ff980089720 tx=0x7ff98c07e210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:54.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.265+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053040 con 0x7ff980083100 2021-03-09T09:56:54.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.265+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c084e80 con 0x7ff980083100 2021-03-09T09:56:54.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.269+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff980083100 2021-03-09T09:56:54.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.269+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:54.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.269+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:54.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:54.269+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:56:56.982 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:56:56.985 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:56:59.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:59.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:59.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ae20 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 rx=0x7ff980089720 tx=0x7ff98c07e210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:59.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ae20 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 rx=0x7ff980089720 tx=0x7ff98c07e210).stop 2021-03-09T09:56:59.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ae20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:59.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ae20 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:59.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:59.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:59.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:59.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011370 con 0x7ff99c0adc20 2021-03-09T09:56:59.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ef20 con 0x7ff99c0613a0 2021-03-09T09:56:59.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009990 con 0x7ff980051540 2021-03-09T09:56:59.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:56:59.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:59.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:56:59.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:59.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:59.273 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:56:59.261+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7567 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:56:59.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:59.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:59.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:59.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:59.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d360 con 0x7ff980051540 2021-03-09T09:56:59.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.261+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980044720 con 0x7ff980051540 2021-03-09T09:56:59.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.265+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7ff98001c8d0 tx=0x7ff994065dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:59.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.265+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980051540 2021-03-09T09:56:59.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.269+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940a9d80 con 0x7ff980051540 2021-03-09T09:56:59.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.269+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:56:59.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.269+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:59.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.269+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:59.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:56:59.269+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:04.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980018dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:04.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980018dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:04.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7ff98001c8d0 tx=0x7ff994065dd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:04.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7ff98001c8d0 tx=0x7ff994065dd0).stop 2021-03-09T09:57:04.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980018dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:04.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:04.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:04.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:04.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:04.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004ba0 con 0x7ff99c0adc20 2021-03-09T09:57:04.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b030 con 0x7ff99c0613a0 2021-03-09T09:57:04.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f260 con 0x7ff980083100 2021-03-09T09:57:04.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:04.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.260+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:04.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.264+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:04.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.264+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:04.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.264+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:04.273 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:04.264+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7612 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:04.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.264+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:04.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.264+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:04.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.264+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:04.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.264+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:04.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.264+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002c4d0 con 0x7ff980083100 2021-03-09T09:57:04.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.264+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800150d0 con 0x7ff980083100 2021-03-09T09:57:04.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.264+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb30 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7ff984040ff0 tx=0x7ff984000cd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:04.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98402e030 con 0x7ff980083100 2021-03-09T09:57:04.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98402e030 con 0x7ff980083100 2021-03-09T09:57:04.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980083100 2021-03-09T09:57:04.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.276+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:04.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.276+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:04.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:04.276+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:09.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:09.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:09.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb30 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7ff984040ff0 tx=0x7ff984000cd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:09.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb30 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7ff984040ff0 tx=0x7ff984000cd0).stop 2021-03-09T09:57:09.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003fb30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:09.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb30 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:09.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:09.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:09.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:09.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f400 con 0x7ff99c0adc20 2021-03-09T09:57:09.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011370 con 0x7ff99c0613a0 2021-03-09T09:57:09.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090770 con 0x7ff980051540 2021-03-09T09:57:09.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:09.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:09.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:09.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:09.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:09.274 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:09.264+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7644 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:09.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98008a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:09.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98008a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:09.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:09.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:09.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980044e20 con 0x7ff980051540 2021-03-09T09:57:09.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800889b0 con 0x7ff980051540 2021-03-09T09:57:09.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.268+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7ff98c00bb30 tx=0x7ff98c00bef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:09.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053030 con 0x7ff980051540 2021-03-09T09:57:09.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c085930 con 0x7ff980051540 2021-03-09T09:57:09.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T09:57:09.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.272+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:09.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.272+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:09.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:09.272+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:14.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980018dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:14.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980018dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:14.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7ff98c00bb30 tx=0x7ff98c00bef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:14.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7ff98c00bb30 tx=0x7ff98c00bef0).stop 2021-03-09T09:57:14.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980018dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:14.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980018dc0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:14.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:14.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:14.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:14.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038550 con 0x7ff99c0adc20 2021-03-09T09:57:14.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003efa0 con 0x7ff99c0613a0 2021-03-09T09:57:14.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:14.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:14.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011370 con 0x7ff980083100 2021-03-09T09:57:14.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.264+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:14.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.268+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:14.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.268+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:14.278 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:14.268+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7681 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:14.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.268+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:14.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.268+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:14.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.268+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:14.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.268+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:14.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.268+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093120 con 0x7ff980083100 2021-03-09T09:57:14.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.268+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002c4d0 con 0x7ff980083100 2021-03-09T09:57:14.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.268+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb00 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff98405aee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:14.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984089ed0 con 0x7ff980083100 2021-03-09T09:57:14.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400dc00 con 0x7ff980083100 2021-03-09T09:57:14.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980083100 2021-03-09T09:57:14.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.276+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:14.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.276+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:14.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:14.276+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:19.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:19.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:19.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb00 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff98405aee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:19.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb00 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff98405aee0).stop 2021-03-09T09:57:19.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003fb00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:19.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fb00 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:19.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:19.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:19.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:19.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038550 con 0x7ff99c0adc20 2021-03-09T09:57:19.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008ab0 con 0x7ff99c0613a0 2021-03-09T09:57:19.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011370 con 0x7ff980051540 2021-03-09T09:57:19.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:19.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:19.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:19.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:19.283 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:19.268+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7716 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:19.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.268+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:19.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:19.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:19.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:19.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:19.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008cd50 con 0x7ff980051540 2021-03-09T09:57:19.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093120 con 0x7ff980051540 2021-03-09T09:57:19.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.276+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984009160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:19.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019070 con 0x7ff980051540 2021-03-09T09:57:19.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400d9d0 con 0x7ff980051540 2021-03-09T09:57:19.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984004600 con 0x7ff980051540 2021-03-09T09:57:19.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.280+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:19.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.280+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:19.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:19.280+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:24.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:24.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:24.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984009160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:24.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984009160).stop 2021-03-09T09:57:24.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:24.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:24.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:24.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:24.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:24.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800126e0 con 0x7ff99c0adc20 2021-03-09T09:57:24.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005650 con 0x7ff99c0613a0 2021-03-09T09:57:24.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010860 con 0x7ff980083100 2021-03-09T09:57:24.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:24.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:24.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:24.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:24.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:24.281 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:24.272+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7754 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:24.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:24.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:24.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:24.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:24.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091ae0 con 0x7ff980083100 2021-03-09T09:57:24.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008cd50 con 0x7ff980083100 2021-03-09T09:57:24.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fab0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7ff9840896c0 tx=0x7ff9840a5070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:24.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019040 con 0x7ff980083100 2021-03-09T09:57:24.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984089500 con 0x7ff980083100 2021-03-09T09:57:24.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984047b10 con 0x7ff980083100 2021-03-09T09:57:24.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.280+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:24.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.280+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:24.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:24.280+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:27.229 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:57:27.232 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:57:29.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:29.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:29.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fab0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7ff9840896c0 tx=0x7ff9840a5070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:29.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fab0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7ff9840896c0 tx=0x7ff9840a5070).stop 2021-03-09T09:57:29.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003fab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:29.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003fab0 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:29.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:29.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:29.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:29.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980031990 con 0x7ff99c0adc20 2021-03-09T09:57:29.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008ab0 con 0x7ff99c0613a0 2021-03-09T09:57:29.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800150d0 con 0x7ff980051540 2021-03-09T09:57:29.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:29.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:29.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:29.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:29.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.272+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:29.283 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:29.272+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7787 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:29.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.276+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:29.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.276+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:29.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.276+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:29.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.276+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:29.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.276+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800292f0 con 0x7ff980051540 2021-03-09T09:57:29.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.276+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091ae0 con 0x7ff980051540 2021-03-09T09:57:29.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.276+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 rx=0x7ff9840449f0 tx=0x7ff984089630).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:29.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984087000 con 0x7ff980051540 2021-03-09T09:57:29.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984047b10 con 0x7ff980051540 2021-03-09T09:57:29.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98407fa10 con 0x7ff980051540 2021-03-09T09:57:29.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.284+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:29.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.284+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:29.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:29.284+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:34.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:34.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:34.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 rx=0x7ff9840449f0 tx=0x7ff984089630).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:34.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 rx=0x7ff9840449f0 tx=0x7ff984089630).stop 2021-03-09T09:57:34.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:34.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:34.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:34.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:34.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:34.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004fac0 con 0x7ff99c0adc20 2021-03-09T09:57:34.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800150d0 con 0x7ff99c0613a0 2021-03-09T09:57:34.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091ae0 con 0x7ff980083100 2021-03-09T09:57:34.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:34.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:34.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:34.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:34.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:34.285 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:34.276+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7819 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:34.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:34.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:34.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:34.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:34.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002c4d0 con 0x7ff980083100 2021-03-09T09:57:34.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980044e20 con 0x7ff980083100 2021-03-09T09:57:34.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.276+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008da60 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 rx=0x7ff98c08ea70 tx=0x7ff98c00ba80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:34.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980083100 2021-03-09T09:57:34.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980083100 2021-03-09T09:57:34.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T09:57:34.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.284+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:34.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.284+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:34.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:34.284+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:39.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:39.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:39.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008da60 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 rx=0x7ff98c08ea70 tx=0x7ff98c00ba80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:39.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008da60 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 rx=0x7ff98c08ea70 tx=0x7ff98c00ba80).stop 2021-03-09T09:57:39.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008da60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:39.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008da60 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:39.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:39.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980008ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:39.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:39.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800150d0 con 0x7ff99c0adc20 2021-03-09T09:57:39.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019790 con 0x7ff99c0613a0 2021-03-09T09:57:39.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091ae0 con 0x7ff980051540 2021-03-09T09:57:39.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:39.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:39.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:39.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:39.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.276+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980008ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:39.287 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:39.276+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7861 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:39.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:39.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:39.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:39.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:39.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027fa0 con 0x7ff980051540 2021-03-09T09:57:39.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002c4d0 con 0x7ff980051540 2021-03-09T09:57:39.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980008ed0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 rx=0x7ff98c0a6fd0 tx=0x7ff98c0a91a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:39.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053080 con 0x7ff980051540 2021-03-09T09:57:39.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980051540 2021-03-09T09:57:39.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff980051540 2021-03-09T09:57:39.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.288+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:39.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.288+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:39.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:39.288+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:44.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980008ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:44.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980008ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:44.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980008ed0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 rx=0x7ff98c0a6fd0 tx=0x7ff98c0a91a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:44.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980008ed0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 rx=0x7ff98c0a6fd0 tx=0x7ff98c0a91a0).stop 2021-03-09T09:57:44.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980008ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:44.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980008ed0 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:44.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:44.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:44.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:44.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177a0 con 0x7ff99c0adc20 2021-03-09T09:57:44.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090f40 con 0x7ff99c0613a0 2021-03-09T09:57:44.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800150d0 con 0x7ff980083100 2021-03-09T09:57:44.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:44.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:44.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:44.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:44.289 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:44.280+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7899 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:44.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:44.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:44.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:44.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:44.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:44.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002c4d0 con 0x7ff980083100 2021-03-09T09:57:44.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.280+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800292f0 con 0x7ff980083100 2021-03-09T09:57:44.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.284+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d720 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 rx=0x7ff98402e4a0 tx=0x7ff9840477e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:44.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98405ae70 con 0x7ff980083100 2021-03-09T09:57:44.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984040e50 con 0x7ff980083100 2021-03-09T09:57:44.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98402e1f0 con 0x7ff980083100 2021-03-09T09:57:44.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.288+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:44.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.288+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:44.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:44.288+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:49.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:57:49.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:49.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d720 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 rx=0x7ff98402e4a0 tx=0x7ff9840477e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:49.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d720 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 rx=0x7ff98402e4a0 tx=0x7ff9840477e0).stop 2021-03-09T09:57:49.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008d720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:49.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d720 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:49.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:49.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:49.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:49.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177a0 con 0x7ff99c0adc20 2021-03-09T09:57:49.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800140a0 con 0x7ff99c0613a0 2021-03-09T09:57:49.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011510 con 0x7ff980051540 2021-03-09T09:57:49.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:49.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:49.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:49.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:49.291 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:49.280+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7925 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:49.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.280+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:49.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:49.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:49.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:49.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:49.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003cac0 con 0x7ff980051540 2021-03-09T09:57:49.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d360 con 0x7ff980051540 2021-03-09T09:57:49.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.288+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940aa9f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:49.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980051540 2021-03-09T09:57:49.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff980051540 2021-03-09T09:57:49.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:57:49.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.292+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:49.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.292+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:49.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:49.292+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:54.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:54.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:54.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940aa9f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:54.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940aa9f0).stop 2021-03-09T09:57:54.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:54.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:54.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:54.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:54.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:54.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980031290 con 0x7ff99c0adc20 2021-03-09T09:57:54.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012110 con 0x7ff99c0613a0 2021-03-09T09:57:54.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177a0 con 0x7ff980083100 2021-03-09T09:57:54.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:54.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:54.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:54.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:54.293 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:54.284+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7971 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:54.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:54.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:54.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:54.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:54.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:54.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091ae0 con 0x7ff980083100 2021-03-09T09:57:54.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027fa0 con 0x7ff980083100 2021-03-09T09:57:54.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.284+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d4b0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 rx=0x7ff980050c70 tx=0x7ff98c07a110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:54.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980083100 2021-03-09T09:57:54.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d030 con 0x7ff980083100 2021-03-09T09:57:54.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980083100 2021-03-09T09:57:54.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.292+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:54.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.292+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:54.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:54.292+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:57:57.300 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:57:57.304 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:57:59.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:59.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:59.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d4b0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 rx=0x7ff980050c70 tx=0x7ff98c07a110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:59.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d4b0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 rx=0x7ff980050c70 tx=0x7ff98c07a110).stop 2021-03-09T09:57:59.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008d4b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:59.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d4b0 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:59.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:59.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:59.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:59.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177a0 con 0x7ff99c0adc20 2021-03-09T09:57:59.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980042ca0 con 0x7ff99c0613a0 2021-03-09T09:57:59.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019790 con 0x7ff980051540 2021-03-09T09:57:59.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:57:59.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:59.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:59.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:57:59.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:59.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:59.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:59.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:59.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:59.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d360 con 0x7ff980051540 2021-03-09T09:57:59.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002c4d0 con 0x7ff980051540 2021-03-09T09:57:59.296 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:57:59.284+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8005 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:57:59.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.284+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008d9a0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 rx=0x7ff9840039a0 tx=0x7ff984044aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:59.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984047080 con 0x7ff980051540 2021-03-09T09:57:59.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984009020 con 0x7ff980051540 2021-03-09T09:57:59.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400f180 con 0x7ff980051540 2021-03-09T09:57:59.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.292+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:59.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.292+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:59.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:57:59.292+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:04.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:04.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:04.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008d9a0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 rx=0x7ff9840039a0 tx=0x7ff984044aa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:04.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008d9a0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 rx=0x7ff9840039a0 tx=0x7ff984044aa0).stop 2021-03-09T09:58:04.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008d9a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:04.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008d9a0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:04.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:04.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:04.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:04.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052c60 con 0x7ff99c0adc20 2021-03-09T09:58:04.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:04.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177a0 con 0x7ff99c0613a0 2021-03-09T09:58:04.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044bb0 con 0x7ff980083100 2021-03-09T09:58:04.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:04.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:04.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:04.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:04.297 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:04.288+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8043 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:04.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:04.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:04.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:04.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:04.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980010b00 con 0x7ff980083100 2021-03-09T09:58:04.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003cac0 con 0x7ff980083100 2021-03-09T09:58:04.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.288+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010610 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7ff99408cfd0 tx=0x7ff994082c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:04.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980083100 2021-03-09T09:58:04.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994035e00 con 0x7ff980083100 2021-03-09T09:58:04.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.296+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T09:58:04.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.296+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:04.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.296+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:04.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:04.296+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:09.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980010610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:58:09.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980010610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:09.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010610 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7ff99408cfd0 tx=0x7ff994082c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:09.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010610 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7ff99408cfd0 tx=0x7ff994082c30).stop 2021-03-09T09:58:09.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980010610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:09.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010610 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:09.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:09.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:09.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:09.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015120 con 0x7ff99c0adc20 2021-03-09T09:58:09.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088c30 con 0x7ff99c0613a0 2021-03-09T09:58:09.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003fc30 con 0x7ff980051540 2021-03-09T09:58:09.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:09.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:09.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:09.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:09.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.288+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:09.299 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:09.288+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8073 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:09.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.292+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:09.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.292+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:09.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.292+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:09.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.292+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:09.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.292+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800098d0 con 0x7ff980051540 2021-03-09T09:58:09.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.292+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d360 con 0x7ff980051540 2021-03-09T09:58:09.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.296+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff9840181e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:09.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.296+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984027120 con 0x7ff980051540 2021-03-09T09:58:09.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.296+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98407e170 con 0x7ff980051540 2021-03-09T09:58:09.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.300+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984089390 con 0x7ff980051540 2021-03-09T09:58:09.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.300+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:09.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.300+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:09.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:09.300+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:14.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.288+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:14.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.288+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:14.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.288+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff9840181e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:14.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.288+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff9840181e0).stop 2021-03-09T09:58:14.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:14.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a3b0 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:14.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:14.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:14.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:14.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088c30 con 0x7ff99c0adc20 2021-03-09T09:58:14.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093120 con 0x7ff99c0613a0 2021-03-09T09:58:14.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012c50 con 0x7ff980083100 2021-03-09T09:58:14.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:14.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:14.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:14.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:14.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:14.305 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:14.292+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8115 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:14.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:14.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:14.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:14.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:14.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002c670 con 0x7ff980083100 2021-03-09T09:58:14.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800098d0 con 0x7ff980083100 2021-03-09T09:58:14.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.292+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008a8a0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 rx=0x7ff98404f5c0 tx=0x7ff98400b570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:14.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.296+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98403f990 con 0x7ff980083100 2021-03-09T09:58:14.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.296+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840270e0 con 0x7ff980083100 2021-03-09T09:58:14.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.300+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984044280 con 0x7ff980083100 2021-03-09T09:58:14.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.300+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:14.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.304+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:14.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:14.304+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:19.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:19.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:19.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008a8a0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 rx=0x7ff98404f5c0 tx=0x7ff98400b570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:19.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008a8a0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 rx=0x7ff98404f5c0 tx=0x7ff98400b570).stop 2021-03-09T09:58:19.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008a8a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:19.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008a8a0 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:19.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:19.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:19.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:19.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088c30 con 0x7ff99c0adc20 2021-03-09T09:58:19.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015120 con 0x7ff99c0613a0 2021-03-09T09:58:19.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177a0 con 0x7ff980051540 2021-03-09T09:58:19.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:19.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:19.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:19.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:19.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:19.304 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:19.291+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8150 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:19.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.295+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:19.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.295+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:19.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.295+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:19.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.295+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:19.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.295+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800098d0 con 0x7ff980051540 2021-03-09T09:58:19.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.295+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002c670 con 0x7ff980051540 2021-03-09T09:58:19.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.295+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052000 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7ff98403fd30 tx=0x7ff98400dad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:19.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840270e0 con 0x7ff980051540 2021-03-09T09:58:19.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98408ee40 con 0x7ff980051540 2021-03-09T09:58:19.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840447a0 con 0x7ff980051540 2021-03-09T09:58:19.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.303+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:19.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.303+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:19.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:19.303+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:24.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980052000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:24.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980052000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:24.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052000 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7ff98403fd30 tx=0x7ff98400dad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:24.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052000 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7ff98403fd30 tx=0x7ff98400dad0).stop 2021-03-09T09:58:24.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980052000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:24.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052000 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:24.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:24.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:24.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:24.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088c30 con 0x7ff99c0adc20 2021-03-09T09:58:24.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033250 con 0x7ff99c0613a0 2021-03-09T09:58:24.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015120 con 0x7ff980083100 2021-03-09T09:58:24.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:24.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:24.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:24.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:24.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.295+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:24.306 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:24.295+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8188 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:24.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:24.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.299+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:24.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:24.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.299+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:24.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002c670 con 0x7ff980083100 2021-03-09T09:58:24.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091ae0 con 0x7ff980083100 2021-03-09T09:58:24.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.299+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c022f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:24.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053090 con 0x7ff980083100 2021-03-09T09:58:24.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980083100 2021-03-09T09:58:24.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980083100 2021-03-09T09:58:24.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.307+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:24.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.307+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:24.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:24.307+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:27.346 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:58:27.349 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:58:29.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:29.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:29.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c022f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:29.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c022f70).stop 2021-03-09T09:58:29.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:29.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:29.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:29.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:29.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:29.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036480 con 0x7ff99c0adc20 2021-03-09T09:58:29.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980096ca0 con 0x7ff99c0613a0 2021-03-09T09:58:29.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088c30 con 0x7ff980051540 2021-03-09T09:58:29.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:29.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:29.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:29.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.295+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:29.308 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:29.299+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8221 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:29.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.299+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:29.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.299+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:29.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.299+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:29.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.299+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:29.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.299+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:29.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.299+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091ae0 con 0x7ff980051540 2021-03-09T09:58:29.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.299+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800098d0 con 0x7ff980051540 2021-03-09T09:58:29.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.303+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984055330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:29.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840a6660 con 0x7ff980051540 2021-03-09T09:58:29.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98407f220 con 0x7ff980051540 2021-03-09T09:58:29.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984000ba0 con 0x7ff980051540 2021-03-09T09:58:29.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.311+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:29.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.311+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:29.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:29.311+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:34.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:34.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:34.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984055330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:34.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984055330).stop 2021-03-09T09:58:34.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:34.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:34.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:34.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:34.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:34.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800488e0 con 0x7ff99c0adc20 2021-03-09T09:58:34.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015120 con 0x7ff99c0613a0 2021-03-09T09:58:34.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088c30 con 0x7ff980083100 2021-03-09T09:58:34.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:34.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:34.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:34.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:34.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:34.310 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:34.299+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8260 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:34.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:34.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:34.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:34.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:34.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980010970 con 0x7ff980083100 2021-03-09T09:58:34.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002c670 con 0x7ff980083100 2021-03-09T09:58:34.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.303+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7ff98c045cc0 tx=0x7ff98c045b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:34.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980083100 2021-03-09T09:58:34.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff980083100 2021-03-09T09:58:34.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980083100 2021-03-09T09:58:34.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.311+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:34.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.311+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:34.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:34.311+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:39.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:39.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:39.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7ff98c045cc0 tx=0x7ff98c045b70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:39.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7ff98c045cc0 tx=0x7ff98c045b70).stop 2021-03-09T09:58:39.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:39.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:39.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:39.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:39.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:39.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033250 con 0x7ff99c0adc20 2021-03-09T09:58:39.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e260 con 0x7ff99c0613a0 2021-03-09T09:58:39.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980032c60 con 0x7ff980051540 2021-03-09T09:58:39.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:39.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:39.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:39.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.299+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:39.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.303+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:39.312 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:39.303+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8293 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:39.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.303+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:39.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.303+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:39.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.303+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:39.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.303+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:39.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.303+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800098d0 con 0x7ff980051540 2021-03-09T09:58:39.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.303+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980010970 con 0x7ff980051540 2021-03-09T09:58:39.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.303+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f0f0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 rx=0x7ff98c005230 tx=0x7ff98c0a2e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:39.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T09:58:39.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d030 con 0x7ff980051540 2021-03-09T09:58:39.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff980051540 2021-03-09T09:58:39.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.311+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:39.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.311+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:39.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:39.311+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:44.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:58:44.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:44.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f0f0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 rx=0x7ff98c005230 tx=0x7ff98c0a2e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:44.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f0f0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 rx=0x7ff98c005230 tx=0x7ff98c0a2e20).stop 2021-03-09T09:58:44.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:44.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f0f0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:44.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:44.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:44.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:44.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800488e0 con 0x7ff99c0adc20 2021-03-09T09:58:44.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015120 con 0x7ff99c0613a0 2021-03-09T09:58:44.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800889b0 con 0x7ff980083100 2021-03-09T09:58:44.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:44.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:44.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:44.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:44.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:44.315 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:44.303+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8331 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:44.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:44.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:44.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:44.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:44.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096fe0 con 0x7ff980083100 2021-03-09T09:58:44.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980010b00 con 0x7ff980083100 2021-03-09T09:58:44.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99407a2a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:44.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069090 con 0x7ff980083100 2021-03-09T09:58:44.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994006c00 con 0x7ff980083100 2021-03-09T09:58:44.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994082dc0 con 0x7ff980083100 2021-03-09T09:58:44.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.315+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:44.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.315+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:44.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:44.315+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:49.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:49.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:49.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99407a2a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:49.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99407a2a0).stop 2021-03-09T09:58:49.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:49.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:49.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:49.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:49.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:49.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800488e0 con 0x7ff99c0adc20 2021-03-09T09:58:49.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177a0 con 0x7ff99c0613a0 2021-03-09T09:58:49.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036480 con 0x7ff980051540 2021-03-09T09:58:49.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:49.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:49.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:49.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:49.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:49.316 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:49.307+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8364 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:49.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:49.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:49.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:49.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:49.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017330 con 0x7ff980051540 2021-03-09T09:58:49.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091ae0 con 0x7ff980051540 2021-03-09T09:58:49.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.307+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003ec10 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840813e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:49.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840555b0 con 0x7ff980051540 2021-03-09T09:58:49.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840555b0 con 0x7ff980051540 2021-03-09T09:58:49.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98402c180 con 0x7ff980051540 2021-03-09T09:58:49.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.319+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:49.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.319+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:49.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:49.319+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:54.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:54.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:54.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003ec10 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840813e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:54.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003ec10 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840813e0).stop 2021-03-09T09:58:54.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003ec10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:54.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003ec10 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:54.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:54.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:54.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:54.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029340 con 0x7ff99c0adc20 2021-03-09T09:58:54.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177a0 con 0x7ff99c0613a0 2021-03-09T09:58:54.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e0d0 con 0x7ff980083100 2021-03-09T09:58:54.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:54.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:54.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:54.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:54.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:54.318 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:54.307+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8395 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:54.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:54.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:54.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:54.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:54.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800132c0 con 0x7ff980083100 2021-03-09T09:58:54.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.307+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800098d0 con 0x7ff980083100 2021-03-09T09:58:54.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 rx=0x7ff98c029ce0 tx=0x7ff98c079820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:54.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980083100 2021-03-09T09:58:54.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c084e90 con 0x7ff980083100 2021-03-09T09:58:54.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980083100 2021-03-09T09:58:54.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.319+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:54.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.319+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:54.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:54.319+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:58:57.459 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:58:57.462 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:58:59.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:59.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:59.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 rx=0x7ff98c029ce0 tx=0x7ff98c079820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:59.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 rx=0x7ff98c029ce0 tx=0x7ff98c079820).stop 2021-03-09T09:58:59.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ac70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:59.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ac70 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:59.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:59.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:59.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:59.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004efc0 con 0x7ff99c0adc20 2021-03-09T09:58:59.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033250 con 0x7ff99c0613a0 2021-03-09T09:58:59.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e240 con 0x7ff980051540 2021-03-09T09:58:59.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:58:59.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:58:59.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:59.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:59.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:59.325 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:58:59.311+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8437 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:58:59.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:59.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:59.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:59.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:59.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800574f0 con 0x7ff980051540 2021-03-09T09:58:59.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800132c0 con 0x7ff980051540 2021-03-09T09:58:59.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.315+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 rx=0x7ff98c08f520 tx=0x7ff98c046eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:59.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T09:58:59.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079fe0 con 0x7ff980051540 2021-03-09T09:58:59.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff980051540 2021-03-09T09:58:59.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.323+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:59.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.323+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:59.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:58:59.323+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:04.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:04.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:04.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 rx=0x7ff98c08f520 tx=0x7ff98c046eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:04.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 rx=0x7ff98c08f520 tx=0x7ff98c046eb0).stop 2021-03-09T09:59:04.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:04.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:04.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:04.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:04.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980097170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:04.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004efc0 con 0x7ff99c0adc20 2021-03-09T09:59:04.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093120 con 0x7ff99c0613a0 2021-03-09T09:59:04.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033250 con 0x7ff980083100 2021-03-09T09:59:04.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:04.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:04.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980097170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:04.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:04.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:04.323 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:04.311+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8476 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:04.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:04.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:04.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:04.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:04.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002c670 con 0x7ff980083100 2021-03-09T09:59:04.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096fe0 con 0x7ff980083100 2021-03-09T09:59:04.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980097170 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 rx=0x7ff980052a20 tx=0x7ff994081420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:04.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069090 con 0x7ff980083100 2021-03-09T09:59:04.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940780d0 con 0x7ff980083100 2021-03-09T09:59:04.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994081af0 con 0x7ff980083100 2021-03-09T09:59:04.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.319+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:04.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.323+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:04.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:04.323+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:09.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980097170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:09.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980097170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:09.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980097170 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 rx=0x7ff980052a20 tx=0x7ff994081420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:09.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980097170 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 rx=0x7ff980052a20 tx=0x7ff994081420).stop 2021-03-09T09:59:09.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980097170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:09.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980097170 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:09.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:09.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:09.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:09.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004efc0 con 0x7ff99c0adc20 2021-03-09T09:59:09.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b870 con 0x7ff99c0613a0 2021-03-09T09:59:09.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018ab0 con 0x7ff980051540 2021-03-09T09:59:09.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:09.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:09.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:09.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:09.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:09.324 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:09.315+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8515 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:09.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:09.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:09.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:09.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:09.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005800 con 0x7ff980051540 2021-03-09T09:59:09.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800574f0 con 0x7ff980051540 2021-03-09T09:59:09.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.315+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 rx=0x7ff98002fc20 tx=0x7ff98c03e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:09.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c053070 con 0x7ff980051540 2021-03-09T09:59:09.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08f360 con 0x7ff980051540 2021-03-09T09:59:09.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T09:59:09.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.323+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:09.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.323+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:09.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:09.323+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:14.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:14.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:14.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 rx=0x7ff98002fc20 tx=0x7ff98c03e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:14.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 rx=0x7ff98002fc20 tx=0x7ff98c03e000).stop 2021-03-09T09:59:14.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:14.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:14.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:14.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:14.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:14.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033250 con 0x7ff99c0adc20 2021-03-09T09:59:14.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004efc0 con 0x7ff99c0613a0 2021-03-09T09:59:14.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e240 con 0x7ff980083100 2021-03-09T09:59:14.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:14.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:14.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:14.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:14.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:14.327 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:14.315+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8539 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:14.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:14.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:14.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:14.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:14.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980090770 con 0x7ff980083100 2021-03-09T09:59:14.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.315+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017330 con 0x7ff980083100 2021-03-09T09:59:14.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.319+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b530 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 rx=0x7ff984081620 tx=0x7ff98404fe90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:14.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984000cd0 con 0x7ff980083100 2021-03-09T09:59:14.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984027020 con 0x7ff980083100 2021-03-09T09:59:14.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98402c180 con 0x7ff980083100 2021-03-09T09:59:14.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.323+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:14.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.327+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:14.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:14.327+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:19.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98001b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:19.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98001b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:19.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b530 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 rx=0x7ff984081620 tx=0x7ff98404fe90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:19.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b530 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 rx=0x7ff984081620 tx=0x7ff98404fe90).stop 2021-03-09T09:59:19.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98001b530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:19.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b530 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:19.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:19.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:19.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:19.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004e100 con 0x7ff99c0adc20 2021-03-09T09:59:19.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018ab0 con 0x7ff99c0613a0 2021-03-09T09:59:19.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036480 con 0x7ff980051540 2021-03-09T09:59:19.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:19.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:19.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:19.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:19.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:19.328 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:19.319+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8577 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:19.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:19.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:19.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:19.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:19.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800132c0 con 0x7ff980051540 2021-03-09T09:59:19.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002c670 con 0x7ff980051540 2021-03-09T09:59:19.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7ff98002c800 tx=0x7ff99401cdd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:19.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069090 con 0x7ff980051540 2021-03-09T09:59:19.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980051540 2021-03-09T09:59:19.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980051540 2021-03-09T09:59:19.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.327+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:19.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.327+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:19.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:19.327+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:24.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:59:24.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:24.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7ff98002c800 tx=0x7ff99401cdd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:24.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7ff98002c800 tx=0x7ff99401cdd0).stop 2021-03-09T09:59:24.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003f370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:24.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003f370 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:24.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:24.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:24.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:24.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003fdd0 con 0x7ff99c0adc20 2021-03-09T09:59:24.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e240 con 0x7ff99c0613a0 2021-03-09T09:59:24.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002af70 con 0x7ff980083100 2021-03-09T09:59:24.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:24.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:24.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:24.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:24.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:24.330 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:24.319+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8619 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:24.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003ff60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:24.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003ff60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:24.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:24.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:24.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091ae0 con 0x7ff980083100 2021-03-09T09:59:24.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800132c0 con 0x7ff980083100 2021-03-09T09:59:24.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b390 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 rx=0x7ff994065e10 tx=0x7ff994074ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:24.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980083100 2021-03-09T09:59:24.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405c990 con 0x7ff980083100 2021-03-09T09:59:24.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940a9c40 con 0x7ff980083100 2021-03-09T09:59:24.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.335+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:24.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.335+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:24.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:24.335+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:27.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:59:27.501 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:59:29.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98001b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:59:29.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98001b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:29.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b390 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 rx=0x7ff994065e10 tx=0x7ff994074ae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:29.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b390 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 rx=0x7ff994065e10 tx=0x7ff994074ae0).stop 2021-03-09T09:59:29.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98001b390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:29.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98001b390 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:29.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:29.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:29.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:29.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:29.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d0a0 con 0x7ff99c0adc20 2021-03-09T09:59:29.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:29.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093120 con 0x7ff99c0613a0 2021-03-09T09:59:29.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002afc0 con 0x7ff980051540 2021-03-09T09:59:29.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:29.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:29.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:29.332 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:29.323+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8647 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:29.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:29.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:29.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:29.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:29.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096fe0 con 0x7ff980051540 2021-03-09T09:59:29.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005800 con 0x7ff980051540 2021-03-09T09:59:29.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097170 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 rx=0x7ff98c0852e0 tx=0x7ff98c00bf60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:29.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075080 con 0x7ff980051540 2021-03-09T09:59:29.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980051540 2021-03-09T09:59:29.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T09:59:29.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.331+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:29.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.331+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:29.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:29.331+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:34.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980097170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:34.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980097170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:34.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097170 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 rx=0x7ff98c0852e0 tx=0x7ff98c00bf60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:34.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097170 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 rx=0x7ff98c0852e0 tx=0x7ff98c00bf60).stop 2021-03-09T09:59:34.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980097170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:34.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097170 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:34.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:34.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980043290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:34.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:34.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044df0 con 0x7ff99c0adc20 2021-03-09T09:59:34.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800060b0 con 0x7ff99c0613a0 2021-03-09T09:59:34.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018f80 con 0x7ff980083100 2021-03-09T09:59:34.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:34.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:34.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:34.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:34.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980043290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:34.335 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:34.323+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8690 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:34.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.327+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:34.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.327+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:34.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.327+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:34.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.327+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:34.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.327+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800132c0 con 0x7ff980083100 2021-03-09T09:59:34.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.327+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096fe0 con 0x7ff980083100 2021-03-09T09:59:34.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.327+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980043290 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7ff98c0a6fd0 tx=0x7ff98c084f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:34.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980083100 2021-03-09T09:59:34.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff980083100 2021-03-09T09:59:34.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff980083100 2021-03-09T09:59:34.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.335+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:34.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.335+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:34.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:34.335+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:39.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980043290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:39.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980043290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:39.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980043290 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7ff98c0a6fd0 tx=0x7ff98c084f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:39.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980043290 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7ff98c0a6fd0 tx=0x7ff98c084f00).stop 2021-03-09T09:59:39.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980043290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:39.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980043290 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:39.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980045110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:39.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:39.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:39.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e240 con 0x7ff99c0adc20 2021-03-09T09:59:39.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093120 con 0x7ff99c0613a0 2021-03-09T09:59:39.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044df0 con 0x7ff980051540 2021-03-09T09:59:39.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:39.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:39.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:39.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:39.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980045110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:39.336 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:39.327+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8724 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:39.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:39.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:39.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:39.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:39.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800111e0 con 0x7ff980051540 2021-03-09T09:59:39.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980090770 con 0x7ff980051540 2021-03-09T09:59:39.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.327+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980045110 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840815a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:39.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840270e0 con 0x7ff980051540 2021-03-09T09:59:39.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984009020 con 0x7ff980051540 2021-03-09T09:59:39.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984044030 con 0x7ff980051540 2021-03-09T09:59:39.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.335+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:39.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.335+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:39.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:39.335+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:44.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980045110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:44.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980045110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:44.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980045110 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840815a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:44.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980045110 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840815a0).stop 2021-03-09T09:59:44.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980045110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:44.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980045110 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:44.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:44.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:44.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:44.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093120 con 0x7ff99c0adc20 2021-03-09T09:59:44.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014fe0 con 0x7ff99c0613a0 2021-03-09T09:59:44.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036480 con 0x7ff980083100 2021-03-09T09:59:44.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:44.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:44.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:44.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:44.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:44.339 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:44.327+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8761 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:44.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:44.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:44.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:44.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:44.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800098d0 con 0x7ff980083100 2021-03-09T09:59:44.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091ae0 con 0x7ff980083100 2021-03-09T09:59:44.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045600 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99407a230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:44.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980083100 2021-03-09T09:59:44.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994035de0 con 0x7ff980083100 2021-03-09T09:59:44.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T09:59:44.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.335+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:44.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.339+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:44.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:44.339+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:49.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980045600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:49.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980045600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:49.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045600 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99407a230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:49.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045600 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99407a230).stop 2021-03-09T09:59:49.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980045600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:49.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045600 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:49.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:49.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:49.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:49.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e240 con 0x7ff99c0adc20 2021-03-09T09:59:49.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044df0 con 0x7ff99c0613a0 2021-03-09T09:59:49.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036480 con 0x7ff980051540 2021-03-09T09:59:49.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:49.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:49.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:49.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:49.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:49.341 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:49.330+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8796 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:49.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:49.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:49.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:49.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:49.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004e440 con 0x7ff980051540 2021-03-09T09:59:49.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800111e0 con 0x7ff980051540 2021-03-09T09:59:49.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.330+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800450b0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7ff98400f460 tx=0x7ff98404fe90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:49.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984019050 con 0x7ff980051540 2021-03-09T09:59:49.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840270e0 con 0x7ff980051540 2021-03-09T09:59:49.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.338+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984044030 con 0x7ff980051540 2021-03-09T09:59:49.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.338+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:49.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.338+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:49.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:49.338+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:54.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800450b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:54.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800450b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:54.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800450b0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7ff98400f460 tx=0x7ff98404fe90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:54.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800450b0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7ff98400f460 tx=0x7ff98404fe90).stop 2021-03-09T09:59:54.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800450b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:54.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800450b0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:54.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:54.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:54.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:54.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088150 con 0x7ff99c0adc20 2021-03-09T09:59:54.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014fe0 con 0x7ff99c0613a0 2021-03-09T09:59:54.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002b5c0 con 0x7ff980083100 2021-03-09T09:59:54.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:54.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:54.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:54.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:54.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:54.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:54.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:54.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:54.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:54.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091ae0 con 0x7ff980083100 2021-03-09T09:59:54.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004e440 con 0x7ff980083100 2021-03-09T09:59:54.345 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:54.334+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8836 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:54.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800455a0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff9840815a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:54.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98404e410 con 0x7ff980083100 2021-03-09T09:59:54.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.338+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98402dbb0 con 0x7ff980083100 2021-03-09T09:59:54.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98404ed00 con 0x7ff980083100 2021-03-09T09:59:54.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.342+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:54.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.342+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:54.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:54.342+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T09:59:57.562 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:59:57.565 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:59:59.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:59.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:59.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800455a0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff9840815a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:59.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800455a0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff9840815a0).stop 2021-03-09T09:59:59.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800455a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:59.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800455a0 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:59.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:59.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:59.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:59.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048310 con 0x7ff99c0adc20 2021-03-09T09:59:59.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044df0 con 0x7ff99c0613a0 2021-03-09T09:59:59.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e0d0 con 0x7ff980051540 2021-03-09T09:59:59.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T09:59:59.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:59.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T09:59:59.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:59.346 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T09:59:59.334+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8867 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T09:59:59.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.334+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:59.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.338+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:59.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.338+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:59.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.338+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:59.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.338+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:59.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.338+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800111e0 con 0x7ff980051540 2021-03-09T09:59:59.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.338+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800098d0 con 0x7ff980051540 2021-03-09T09:59:59.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.338+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044ff0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7ff99408cc40 tx=0x7ff99400b030).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:59.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T09:59:59.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408ccb0 con 0x7ff980051540 2021-03-09T09:59:59.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T09:59:59.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.346+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:59.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.346+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:59.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T09:59:59.346+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:04.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980044ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:04.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980044ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:04.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044ff0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7ff99408cc40 tx=0x7ff99400b030).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:04.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044ff0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7ff99408cc40 tx=0x7ff99400b030).stop 2021-03-09T10:00:04.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980044ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:04.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044ff0 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:04.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:04.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:04.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:04.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e240 con 0x7ff99c0adc20 2021-03-09T10:00:04.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030180 con 0x7ff99c0613a0 2021-03-09T10:00:04.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048310 con 0x7ff980083100 2021-03-09T10:00:04.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:04.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:04.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:04.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.334+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:04.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.338+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:04.349 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:04.338+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8907 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:04.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:04.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.342+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:04.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:04.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.342+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:04.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980090770 con 0x7ff980083100 2021-03-09T10:00:04.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800111e0 con 0x7ff980083100 2021-03-09T10:00:04.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.342+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096780 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 rx=0x7ff98008dba0 tx=0x7ff99409b020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:04.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980083100 2021-03-09T10:00:04.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408ce80 con 0x7ff980083100 2021-03-09T10:00:04.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99406f000 con 0x7ff980083100 2021-03-09T10:00:04.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.346+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:04.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.346+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:04.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:04.346+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:09.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:00:09.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:09.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096780 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 rx=0x7ff98008dba0 tx=0x7ff99409b020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:09.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096780 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 rx=0x7ff98008dba0 tx=0x7ff99409b020).stop 2021-03-09T10:00:09.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:09.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096780 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:09.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:09.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:09.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:09.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e240 con 0x7ff99c0adc20 2021-03-09T10:00:09.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008a810 con 0x7ff99c0613a0 2021-03-09T10:00:09.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048310 con 0x7ff980051540 2021-03-09T10:00:09.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:09.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:09.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:09.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:09.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.338+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:09.351 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:09.338+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8945 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:09.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:09.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:09.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:09.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:09.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980006250 con 0x7ff980051540 2021-03-09T10:00:09.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800132c0 con 0x7ff980051540 2021-03-09T10:00:09.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 rx=0x7ff98c07efe0 tx=0x7ff98c084f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:09.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075090 con 0x7ff980051540 2021-03-09T10:00:09.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d080 con 0x7ff980051540 2021-03-09T10:00:09.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T10:00:09.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.346+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:09.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.350+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:09.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:09.350+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:14.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980044fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:14.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980044fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:14.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 rx=0x7ff98c07efe0 tx=0x7ff98c084f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:14.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 rx=0x7ff98c07efe0 tx=0x7ff98c084f70).stop 2021-03-09T10:00:14.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980044fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:14.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:14.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:14.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:14.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:14.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b1b0 con 0x7ff99c0adc20 2021-03-09T10:00:14.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800332a0 con 0x7ff99c0613a0 2021-03-09T10:00:14.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044df0 con 0x7ff980083100 2021-03-09T10:00:14.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:14.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:14.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:14.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:14.352 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:14.342+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8980 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:14.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:14.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:14.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:14.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:14.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:14.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800132c0 con 0x7ff980083100 2021-03-09T10:00:14.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.342+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091ae0 con 0x7ff980083100 2021-03-09T10:00:14.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.346+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984000ce0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:14.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984081400 con 0x7ff980083100 2021-03-09T10:00:14.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984081400 con 0x7ff980083100 2021-03-09T10:00:14.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980083100 2021-03-09T10:00:14.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.354+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:14.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.354+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:14.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:14.354+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:19.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980018ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:19.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980018ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:19.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984000ce0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:19.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984000ce0).stop 2021-03-09T10:00:19.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980018ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:19.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:19.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:19.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:19.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:19.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b350 con 0x7ff99c0adc20 2021-03-09T10:00:19.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800319b0 con 0x7ff99c0613a0 2021-03-09T10:00:19.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044df0 con 0x7ff980051540 2021-03-09T10:00:19.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:19.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:19.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:19.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:19.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:19.355 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:19.342+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9012 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:19.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:19.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:19.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:19.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:19.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800098d0 con 0x7ff980051540 2021-03-09T10:00:19.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980006250 con 0x7ff980051540 2021-03-09T10:00:19.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.346+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7ff98001b6a0 tx=0x7ff98c0991d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:19.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T10:00:19.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980051540 2021-03-09T10:00:19.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T10:00:19.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.350+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:19.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.350+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:19.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:19.350+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:24.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980044fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:24.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980044fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:24.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7ff98001b6a0 tx=0x7ff98c0991d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:24.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7ff98001b6a0 tx=0x7ff98c0991d0).stop 2021-03-09T10:00:24.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980044fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:24.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980044fa0 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:24.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:24.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:24.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:24.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001e650 con 0x7ff99c0adc20 2021-03-09T10:00:24.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800930f0 con 0x7ff99c0613a0 2021-03-09T10:00:24.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022a90 con 0x7ff980083100 2021-03-09T10:00:24.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:24.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:24.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:24.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:24.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:24.356 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:24.346+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9052 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:24.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:24.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:24.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:24.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:24.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088490 con 0x7ff980083100 2021-03-09T10:00:24.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800098d0 con 0x7ff980083100 2021-03-09T10:00:24.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.350+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 rx=0x7ff98c098040 tx=0x7ff98c08ee20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:24.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980083100 2021-03-09T10:00:24.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07d0f0 con 0x7ff980083100 2021-03-09T10:00:24.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff980083100 2021-03-09T10:00:24.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.354+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:24.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.358+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:24.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:24.358+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:27.609 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:00:27.617 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:00:29.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980018ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:00:29.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980018ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:29.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 rx=0x7ff98c098040 tx=0x7ff98c08ee20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:29.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 rx=0x7ff98c098040 tx=0x7ff98c08ee20).stop 2021-03-09T10:00:29.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980018ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:29.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980018ab0 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:29.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:29.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:29.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:29.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980040d80 con 0x7ff99c0adc20 2021-03-09T10:00:29.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006250 con 0x7ff99c0613a0 2021-03-09T10:00:29.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800111e0 con 0x7ff980051540 2021-03-09T10:00:29.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:29.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:29.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:29.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:29.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:29.358 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:29.350+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9077 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:29.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:29.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:29.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:29.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:29.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027fa0 con 0x7ff980051540 2021-03-09T10:00:29.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088490 con 0x7ff980051540 2021-03-09T10:00:29.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.350+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7ff98c03d300 tx=0x7ff98c08ecb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:29.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c03d140 con 0x7ff980051540 2021-03-09T10:00:29.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07ed10 con 0x7ff980051540 2021-03-09T10:00:29.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff980051540 2021-03-09T10:00:29.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.358+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:29.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.358+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:29.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:29.358+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:34.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980010cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:34.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980010cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:34.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7ff98c03d300 tx=0x7ff98c08ecb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:34.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7ff98c03d300 tx=0x7ff98c08ecb0).stop 2021-03-09T10:00:34.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:34.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:34.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:34.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:34.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:34.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038410 con 0x7ff99c0adc20 2021-03-09T10:00:34.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800930f0 con 0x7ff99c0613a0 2021-03-09T10:00:34.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800340a0 con 0x7ff980083100 2021-03-09T10:00:34.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:34.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.346+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:34.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:34.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.350+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:34.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.350+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:34.361 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:34.350+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9121 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:34.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:34.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:34.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:34.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:34.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800111e0 con 0x7ff980083100 2021-03-09T10:00:34.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980090770 con 0x7ff980083100 2021-03-09T10:00:34.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045290 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7ff994077060 tx=0x7ff99403db40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:34.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff980083100 2021-03-09T10:00:34.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff980083100 2021-03-09T10:00:34.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:00:34.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.362+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:34.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.362+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:34.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:34.362+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:39.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980045290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:39.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980045290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:39.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045290 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7ff994077060 tx=0x7ff99403db40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:39.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045290 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7ff994077060 tx=0x7ff99403db40).stop 2021-03-09T10:00:39.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980045290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:39.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980045290 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:39.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:39.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:39.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:39.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008dce0 con 0x7ff99c0adc20 2021-03-09T10:00:39.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d480 con 0x7ff99c0613a0 2021-03-09T10:00:39.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800098d0 con 0x7ff980051540 2021-03-09T10:00:39.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:39.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:39.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003cf90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:39.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:39.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:39.364 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:39.354+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9163 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:39.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:39.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:39.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003cf90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:39.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:39.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980006250 con 0x7ff980051540 2021-03-09T10:00:39.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027fa0 con 0x7ff980051540 2021-03-09T10:00:39.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7ff98c084f40 tx=0x7ff98c04bfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:39.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075090 con 0x7ff980051540 2021-03-09T10:00:39.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d080 con 0x7ff980051540 2021-03-09T10:00:39.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:00:39.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.358+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:39.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.362+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:39.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:39.362+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:44.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980010cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:44.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980010cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:44.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7ff98c084f40 tx=0x7ff98c04bfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:44.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7ff98c084f40 tx=0x7ff98c04bfc0).stop 2021-03-09T10:00:44.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:44.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980010cf0 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:44.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:44.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:44.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:44.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003efc0 con 0x7ff99c0adc20 2021-03-09T10:00:44.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800957a0 con 0x7ff99c0613a0 2021-03-09T10:00:44.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043ce0 con 0x7ff980083100 2021-03-09T10:00:44.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:44.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:44.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003cf90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:44.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:44.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:44.367 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:44.354+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9191 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:44.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:44.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:44.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003cf90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:44.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:44.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088490 con 0x7ff980083100 2021-03-09T10:00:44.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800111e0 con 0x7ff980083100 2021-03-09T10:00:44.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004e2a0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7ff99403dc20 tx=0x7ff9940318d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:44.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980083100 2021-03-09T10:00:44.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994007e20 con 0x7ff980083100 2021-03-09T10:00:44.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994081af0 con 0x7ff980083100 2021-03-09T10:00:44.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.366+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:44.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.366+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:44.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:44.366+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:49.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:49.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:49.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004e2a0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7ff99403dc20 tx=0x7ff9940318d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:49.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004e2a0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7ff99403dc20 tx=0x7ff9940318d0).stop 2021-03-09T10:00:49.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004e2a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:49.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004e2a0 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:49.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:49.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:49.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:49.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003cf90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:49.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003efc0 con 0x7ff99c0adc20 2021-03-09T10:00:49.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:49.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052e90 con 0x7ff99c0613a0 2021-03-09T10:00:49.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016940 con 0x7ff980051540 2021-03-09T10:00:49.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:49.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:49.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:49.368 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:49.358+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9229 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:49.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:49.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:49.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003cf90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:49.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:49.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800098d0 con 0x7ff980051540 2021-03-09T10:00:49.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088490 con 0x7ff980051540 2021-03-09T10:00:49.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095e20 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7ff994081de0 tx=0x7ff9940780e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:49.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405d2a0 con 0x7ff980051540 2021-03-09T10:00:49.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405d2a0 con 0x7ff980051540 2021-03-09T10:00:49.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940a9c10 con 0x7ff980051540 2021-03-09T10:00:49.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.366+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:49.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.366+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:49.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:49.366+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:54.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:54.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:54.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095e20 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7ff994081de0 tx=0x7ff9940780e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:54.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095e20 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7ff994081de0 tx=0x7ff9940780e0).stop 2021-03-09T10:00:54.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:54.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095e20 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:54.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:54.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:54.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:54.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002f650 con 0x7ff99c0adc20 2021-03-09T10:00:54.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016940 con 0x7ff99c0613a0 2021-03-09T10:00:54.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c840 con 0x7ff980083100 2021-03-09T10:00:54.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003ce00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:54.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:54.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:54.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:54.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:54.371 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:54.362+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9271 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:54.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:54.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:54.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003ce00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:54.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:54.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002ff70 con 0x7ff980083100 2021-03-09T10:00:54.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800132c0 con 0x7ff980083100 2021-03-09T10:00:54.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.362+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010cf0 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7ff9840355a0 tx=0x7ff98402e810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:54.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984009020 con 0x7ff980083100 2021-03-09T10:00:54.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400b4b0 con 0x7ff980083100 2021-03-09T10:00:54.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400f180 con 0x7ff980083100 2021-03-09T10:00:54.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.370+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:54.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.370+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:54.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:54.370+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:00:57.747 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:00:57.756 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:00:59.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980010cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:59.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980010cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:59.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010cf0 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7ff9840355a0 tx=0x7ff98402e810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:59.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010cf0 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7ff9840355a0 tx=0x7ff98402e810).stop 2021-03-09T10:00:59.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:59.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980010cf0 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:59.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:59.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:59.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:59.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016d90 con 0x7ff99c0adc20 2021-03-09T10:00:59.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003ee00 con 0x7ff99c0613a0 2021-03-09T10:00:59.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d7d0 con 0x7ff980051540 2021-03-09T10:00:59.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:59.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003ce00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:00:59.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:59.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:00:59.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:59.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:59.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:59.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003ce00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:59.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:59.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027fa0 con 0x7ff980051540 2021-03-09T10:00:59.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002ff70 con 0x7ff980051540 2021-03-09T10:00:59.374 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:00:59.362+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9314 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:00:59.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.362+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840477e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:59.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984047120 con 0x7ff980051540 2021-03-09T10:00:59.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984047280 con 0x7ff980051540 2021-03-09T10:00:59.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98404fc30 con 0x7ff980051540 2021-03-09T10:00:59.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.370+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:59.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.370+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:59.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:00:59.370+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:04.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:04.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:04.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840477e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:04.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840477e0).stop 2021-03-09T10:01:04.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:04.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:04.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:04.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:04.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:04.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045480 con 0x7ff99c0adc20 2021-03-09T10:01:04.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003ce00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:04.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016d40 con 0x7ff99c0613a0 2021-03-09T10:01:04.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002ff70 con 0x7ff980083100 2021-03-09T10:01:04.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:04.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:04.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:04.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:04.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:04.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:04.380 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:04.366+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9347 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:04.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003ce00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:04.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:04.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980011030 con 0x7ff980083100 2021-03-09T10:01:04.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800098d0 con 0x7ff980083100 2021-03-09T10:01:04.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.366+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d2f0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7ff99407bc60 tx=0x7ff9940114b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:04.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980083100 2021-03-09T10:01:04.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407be60 con 0x7ff980083100 2021-03-09T10:01:04.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99405d940 con 0x7ff980083100 2021-03-09T10:01:04.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.374+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:04.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.374+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:04.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:04.374+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:09.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:09.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:09.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d2f0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7ff99407bc60 tx=0x7ff9940114b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:09.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d2f0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7ff99407bc60 tx=0x7ff9940114b0).stop 2021-03-09T10:01:09.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003d2f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:09.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d2f0 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:09.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:09.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:09.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:09.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019b20 con 0x7ff99c0adc20 2021-03-09T10:01:09.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800340a0 con 0x7ff99c0613a0 2021-03-09T10:01:09.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004b50 con 0x7ff980051540 2021-03-09T10:01:09.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003ce00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:09.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:09.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:09.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:09.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:09.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:09.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:09.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff98003ce00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:09.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff98003ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:09.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002ff70 con 0x7ff980051540 2021-03-09T10:01:09.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980011030 con 0x7ff980051540 2021-03-09T10:01:09.380 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:09.370+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9382 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:09.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7ff98008a920 tx=0x7ff994009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:09.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081bc0 con 0x7ff980051540 2021-03-09T10:01:09.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940a9c10 con 0x7ff980051540 2021-03-09T10:01:09.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940a9c10 con 0x7ff980051540 2021-03-09T10:01:09.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.374+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:09.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.374+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:09.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:09.374+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:14.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:14.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:14.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7ff98008a920 tx=0x7ff994009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:14.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7ff98008a920 tx=0x7ff994009a70).stop 2021-03-09T10:01:14.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:14.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:14.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:14.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:14.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:14.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c840 con 0x7ff99c0adc20 2021-03-09T10:01:14.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057a30 con 0x7ff99c0613a0 2021-03-09T10:01:14.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800111e0 con 0x7ff980083100 2021-03-09T10:01:14.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:14.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:14.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:14.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:14.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:14.381 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:14.370+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9418 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:14.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:14.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:14.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:14.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:14.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003d2f0 con 0x7ff980083100 2021-03-09T10:01:14.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002ff70 con 0x7ff980083100 2021-03-09T10:01:14.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003ce00 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 rx=0x7ff99400c010 tx=0x7ff99400bed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:14.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994097e70 con 0x7ff980083100 2021-03-09T10:01:14.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.382+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994097e70 con 0x7ff980083100 2021-03-09T10:01:14.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.382+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994081af0 con 0x7ff980083100 2021-03-09T10:01:14.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.382+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:14.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.382+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:14.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:14.382+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:19.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:19.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:19.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003ce00 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 rx=0x7ff99400c010 tx=0x7ff99400bed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:19.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003ce00 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 rx=0x7ff99400c010 tx=0x7ff99400bed0).stop 2021-03-09T10:01:19.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003ce00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:19.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003ce00 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:19.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:19.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:19.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:19.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002f7f0 con 0x7ff99c0adc20 2021-03-09T10:01:19.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e120 con 0x7ff99c0613a0 2021-03-09T10:01:19.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043e80 con 0x7ff980051540 2021-03-09T10:01:19.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:19.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:19.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:19.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:19.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:19.383 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:19.374+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9445 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:19.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:19.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:19.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:19.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:19.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002ff70 con 0x7ff980051540 2021-03-09T10:01:19.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.374+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027fa0 con 0x7ff980051540 2021-03-09T10:01:19.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.378+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984047a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:19.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984009020 con 0x7ff980051540 2021-03-09T10:01:19.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984009020 con 0x7ff980051540 2021-03-09T10:01:19.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.382+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98404fba0 con 0x7ff980051540 2021-03-09T10:01:19.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.382+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:19.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.382+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:19.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:19.382+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:24.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:24.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:24.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984047a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:24.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984047a30).stop 2021-03-09T10:01:24.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980095de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:24.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980095de0 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:24.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:24.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:24.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:24.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a170 con 0x7ff99c0adc20 2021-03-09T10:01:24.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002f990 con 0x7ff99c0613a0 2021-03-09T10:01:24.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018ca0 con 0x7ff980083100 2021-03-09T10:01:24.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:24.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:24.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:24.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:24.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:24.385 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:24.374+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9486 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:24.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:24.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:24.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:24.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:24.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009a70 con 0x7ff980083100 2021-03-09T10:01:24.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980006250 con 0x7ff980083100 2021-03-09T10:01:24.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.378+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd90 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7ff98c0a02f0 tx=0x7ff98c079340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:24.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980083100 2021-03-09T10:01:24.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980083100 2021-03-09T10:01:24.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.382+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:01:24.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.382+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:24.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.382+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:24.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:24.382+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:27.819 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:01:27.822 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:01:29.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:29.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:29.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd90 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7ff98c0a02f0 tx=0x7ff98c079340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:29.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd90 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7ff98c0a02f0 tx=0x7ff98c079340).stop 2021-03-09T10:01:29.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003cd90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:29.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd90 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:29.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:29.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:29.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:29.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088390 con 0x7ff99c0adc20 2021-03-09T10:01:29.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043e80 con 0x7ff99c0613a0 2021-03-09T10:01:29.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006250 con 0x7ff980051540 2021-03-09T10:01:29.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:29.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:29.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:29.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:29.388 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:29.377+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9517 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:29.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:29.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:29.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:29.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:29.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:29.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027fa0 con 0x7ff980051540 2021-03-09T10:01:29.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002ff70 con 0x7ff980051540 2021-03-09T10:01:29.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.377+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 rx=0x7ff98402d3a0 tx=0x7ff984009020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:29.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.381+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984044390 con 0x7ff980051540 2021-03-09T10:01:29.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984044390 con 0x7ff980051540 2021-03-09T10:01:29.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980051540 2021-03-09T10:01:29.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.385+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:29.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.385+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:29.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:29.385+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:34.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800194f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:34.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800194f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:34.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 rx=0x7ff98402d3a0 tx=0x7ff984009020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:34.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 rx=0x7ff98402d3a0 tx=0x7ff984009020).stop 2021-03-09T10:01:34.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800194f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:34.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:34.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:34.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:34.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:34.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e620 con 0x7ff99c0adc20 2021-03-09T10:01:34.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002b5f0 con 0x7ff99c0613a0 2021-03-09T10:01:34.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002f990 con 0x7ff980083100 2021-03-09T10:01:34.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:34.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:34.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:34.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:34.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:34.390 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:34.377+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9565 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:34.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:34.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:34.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:34.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:34.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980095de0 con 0x7ff980083100 2021-03-09T10:01:34.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.377+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003d2f0 con 0x7ff980083100 2021-03-09T10:01:34.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.381+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd40 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 rx=0x7ff98008da70 tx=0x7ff994081670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:34.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.381+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081af0 con 0x7ff980083100 2021-03-09T10:01:34.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405c990 con 0x7ff980083100 2021-03-09T10:01:34.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070080 con 0x7ff980083100 2021-03-09T10:01:34.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.385+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:34.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.389+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:34.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:34.389+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:39.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:39.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:39.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd40 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 rx=0x7ff98008da70 tx=0x7ff994081670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:39.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd40 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 rx=0x7ff98008da70 tx=0x7ff994081670).stop 2021-03-09T10:01:39.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003cd40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:39.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd40 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:39.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:39.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:39.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:39.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800122b0 con 0x7ff99c0adc20 2021-03-09T10:01:39.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800957f0 con 0x7ff99c0613a0 2021-03-09T10:01:39.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800199e0 con 0x7ff980051540 2021-03-09T10:01:39.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:39.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:39.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:39.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:39.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:39.392 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:39.381+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9596 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:39.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:39.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:39.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:39.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:39.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980095f70 con 0x7ff980051540 2021-03-09T10:01:39.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980095de0 con 0x7ff980051540 2021-03-09T10:01:39.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.381+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99401cdd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:39.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980051540 2021-03-09T10:01:39.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.389+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994035470 con 0x7ff980051540 2021-03-09T10:01:39.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994078000 con 0x7ff980051540 2021-03-09T10:01:39.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.393+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:39.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.393+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:39.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:39.393+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:44.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800194f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:44.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800194f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:44.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99401cdd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:44.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99401cdd0).stop 2021-03-09T10:01:44.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800194f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:44.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800194f0 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:44.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:44.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:44.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:44.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800065d0 con 0x7ff99c0adc20 2021-03-09T10:01:44.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050380 con 0x7ff99c0613a0 2021-03-09T10:01:44.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:44.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094fd0 con 0x7ff980083100 2021-03-09T10:01:44.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:44.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:44.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:44.395 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:44.385+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9624 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:44.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:44.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:44.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:44.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:44.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:44.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980095de0 con 0x7ff980083100 2021-03-09T10:01:44.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.385+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027fa0 con 0x7ff980083100 2021-03-09T10:01:44.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.389+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7ff984058d70 tx=0x7ff984015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:44.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984027080 con 0x7ff980083100 2021-03-09T10:01:44.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984027080 con 0x7ff980083100 2021-03-09T10:01:44.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984015b00 con 0x7ff980083100 2021-03-09T10:01:44.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.397+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:44.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.397+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:44.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:44.397+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:49.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980024ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:49.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980024ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:49.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7ff984058d70 tx=0x7ff984015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:49.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7ff984058d70 tx=0x7ff984015fa0).stop 2021-03-09T10:01:49.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980024ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:49.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:49.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:49.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:49.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:49.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f120 con 0x7ff99c0adc20 2021-03-09T10:01:49.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008800 con 0x7ff99c0613a0 2021-03-09T10:01:49.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800555d0 con 0x7ff980051540 2021-03-09T10:01:49.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:49.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:49.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:49.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:49.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:49.397 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:49.385+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9661 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:49.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:49.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:49.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:49.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:49.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980044020 con 0x7ff980051540 2021-03-09T10:01:49.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.385+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980095de0 con 0x7ff980051540 2021-03-09T10:01:49.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.389+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7ff98407b790 tx=0x7ff98404e4d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:49.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.389+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984047aa0 con 0x7ff980051540 2021-03-09T10:01:49.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.389+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984003900 con 0x7ff980051540 2021-03-09T10:01:49.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984015730 con 0x7ff980051540 2021-03-09T10:01:49.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.393+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:49.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.393+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:49.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:49.393+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:54.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:01:54.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:54.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7ff98407b790 tx=0x7ff98404e4d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:54.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7ff98407b790 tx=0x7ff98404e4d0).stop 2021-03-09T10:01:54.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:54.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:54.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:54.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:54.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:54.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f120 con 0x7ff99c0adc20 2021-03-09T10:01:54.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055770 con 0x7ff99c0613a0 2021-03-09T10:01:54.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800065d0 con 0x7ff980083100 2021-03-09T10:01:54.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:54.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:54.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:54.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:54.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:54.399 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:54.389+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9696 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:54.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:54.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:54.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:54.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:54.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800132c0 con 0x7ff980083100 2021-03-09T10:01:54.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009a70 con 0x7ff980083100 2021-03-09T10:01:54.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d080 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c006a80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:54.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075090 con 0x7ff980083100 2021-03-09T10:01:54.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.389+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d080 con 0x7ff980083100 2021-03-09T10:01:54.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:01:54.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.393+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:54.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.397+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:54.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:54.397+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:01:57.865 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:01:57.868 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:01:59.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:59.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:59.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d080 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c006a80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:59.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d080 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c006a80).stop 2021-03-09T10:01:59.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003d080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:59.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d080 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:59.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:59.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:59.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:59.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800499c0 con 0x7ff99c0adc20 2021-03-09T10:01:59.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f120 con 0x7ff99c0613a0 2021-03-09T10:01:59.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008b320 con 0x7ff980051540 2021-03-09T10:01:59.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:01:59.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:59.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:59.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:01:59.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:59.402 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:01:59.393+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9733 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:01:59.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:59.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:59.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:59.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:59.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045870 con 0x7ff980051540 2021-03-09T10:01:59.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980044020 con 0x7ff980051540 2021-03-09T10:01:59.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.393+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055270 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7ff984000da0 tx=0x7ff984046290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:59.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984035420 con 0x7ff980051540 2021-03-09T10:01:59.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984027540 con 0x7ff980051540 2021-03-09T10:01:59.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.401+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980051540 2021-03-09T10:01:59.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.401+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:59.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.405+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:59.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:01:59.405+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:04.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980055270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:04.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980055270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:04.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055270 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7ff984000da0 tx=0x7ff984046290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:04.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055270 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7ff984000da0 tx=0x7ff984046290).stop 2021-03-09T10:02:04.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980055270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:04.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055270 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:04.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:04.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:04.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:04.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002b790 con 0x7ff99c0adc20 2021-03-09T10:02:04.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800907c0 con 0x7ff99c0613a0 2021-03-09T10:02:04.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800491b0 con 0x7ff980083100 2021-03-09T10:02:04.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:04.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:04.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:04.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:04.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.393+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:04.403 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:04.397+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9768 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:04.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.397+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:04.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.397+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:04.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.397+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:04.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.397+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:04.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.397+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980095de0 con 0x7ff980083100 2021-03-09T10:02:04.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.397+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800132c0 con 0x7ff980083100 2021-03-09T10:02:04.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.397+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd10 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c079700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:04.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.401+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980083100 2021-03-09T10:02:04.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980083100 2021-03-09T10:02:04.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980083100 2021-03-09T10:02:04.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.405+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:04.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.405+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:04.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:04.405+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:09.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:09.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:09.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd10 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c079700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:09.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd10 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c079700).stop 2021-03-09T10:02:09.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003cd10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:09.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003cd10 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:09.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:09.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:09.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:09.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800184d0 con 0x7ff99c0adc20 2021-03-09T10:02:09.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016ae0 con 0x7ff99c0613a0 2021-03-09T10:02:09.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001e7f0 con 0x7ff980051540 2021-03-09T10:02:09.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:09.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:09.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:09.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:09.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.397+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:09.406 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:09.397+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9805 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:09.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:09.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.401+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:09.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:09.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.401+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:09.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800132c0 con 0x7ff980051540 2021-03-09T10:02:09.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980095de0 con 0x7ff980051540 2021-03-09T10:02:09.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.401+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055030 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7ff98c009ad0 tx=0x7ff98c08dc00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:09.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075080 con 0x7ff980051540 2021-03-09T10:02:09.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980051540 2021-03-09T10:02:09.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff980051540 2021-03-09T10:02:09.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.409+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:09.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.409+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:09.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:09.409+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:14.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980055030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:02:14.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980055030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:14.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055030 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7ff98c009ad0 tx=0x7ff98c08dc00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:14.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055030 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7ff98c009ad0 tx=0x7ff98c08dc00).stop 2021-03-09T10:02:14.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980055030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:14.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980055030 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:14.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:14.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:14.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:14.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049d00 con 0x7ff99c0adc20 2021-03-09T10:02:14.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e670 con 0x7ff99c0613a0 2021-03-09T10:02:14.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050370 con 0x7ff980083100 2021-03-09T10:02:14.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:14.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:14.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:14.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:14.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.397+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:14.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:14.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.401+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:14.409 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:14.401+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9839 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:14.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:14.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.401+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:14.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000a8e0 con 0x7ff980083100 2021-03-09T10:02:14.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800132c0 con 0x7ff980083100 2021-03-09T10:02:14.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.401+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d010 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c006650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:14.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.401+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980083100 2021-03-09T10:02:14.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.401+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff980083100 2021-03-09T10:02:14.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980083100 2021-03-09T10:02:14.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.405+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:14.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.409+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:14.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:14.409+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:19.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:02:19.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:19.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d010 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c006650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:19.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d010 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c006650).stop 2021-03-09T10:02:19.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003d010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:19.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003d010 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:19.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:19.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:19.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:19.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049d00 con 0x7ff99c0adc20 2021-03-09T10:02:19.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012300 con 0x7ff99c0613a0 2021-03-09T10:02:19.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055710 con 0x7ff980051540 2021-03-09T10:02:19.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:19.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:19.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:19.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:19.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:19.410 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:19.401+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9878 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:19.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:19.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:19.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:19.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:19.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800199d0 con 0x7ff980051540 2021-03-09T10:02:19.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.401+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980095f70 con 0x7ff980051540 2021-03-09T10:02:19.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.405+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99403da60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:19.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:02:19.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff980051540 2021-03-09T10:02:19.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:02:19.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.409+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:19.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.413+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:19.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:19.413+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:24.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.401+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:24.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.401+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:24.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.401+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99403da60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:24.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.401+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99403da60).stop 2021-03-09T10:02:24.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.401+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:24.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.401+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:24.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.401+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:24.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.401+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:24.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:24.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ef20 con 0x7ff99c0adc20 2021-03-09T10:02:24.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056140 con 0x7ff99c0613a0 2021-03-09T10:02:24.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049d00 con 0x7ff980083100 2021-03-09T10:02:24.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:24.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:24.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:24.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:24.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:24.413 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:24.405+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9912 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:24.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:24.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:24.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:24.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:24.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009a70 con 0x7ff980083100 2021-03-09T10:02:24.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.405+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045870 con 0x7ff980083100 2021-03-09T10:02:24.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.409+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff98402ae40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:24.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984009070 con 0x7ff980083100 2021-03-09T10:02:24.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980083100 2021-03-09T10:02:24.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b410 con 0x7ff980083100 2021-03-09T10:02:24.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.413+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:24.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.417+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:24.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:24.417+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:27.927 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:02:27.931 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:02:29.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:29.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:29.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff98402ae40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:29.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff98402ae40).stop 2021-03-09T10:02:29.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:29.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:29.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:29.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:29.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:29.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800432f0 con 0x7ff99c0adc20 2021-03-09T10:02:29.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce00 con 0x7ff99c0613a0 2021-03-09T10:02:29.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980040f80 con 0x7ff980051540 2021-03-09T10:02:29.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:29.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:29.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:29.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:29.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.405+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:29.416 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:29.405+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9949 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:29.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.409+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:29.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.409+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:29.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.409+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:29.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.409+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:29.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.409+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002bad0 con 0x7ff980051540 2021-03-09T10:02:29.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.409+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000a8e0 con 0x7ff980051540 2021-03-09T10:02:29.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.409+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 rx=0x7ff98c07e660 tx=0x7ff98c07df80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:29.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0840e0 con 0x7ff980051540 2021-03-09T10:02:29.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d090 con 0x7ff980051540 2021-03-09T10:02:29.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:02:29.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.413+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:29.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.417+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:29.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:29.417+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:34.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:34.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:34.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 rx=0x7ff98c07e660 tx=0x7ff98c07df80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:34.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 rx=0x7ff98c07e660 tx=0x7ff98c07df80).stop 2021-03-09T10:02:34.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:34.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:34.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:34.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:34.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:34.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013cf0 con 0x7ff99c0adc20 2021-03-09T10:02:34.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014190 con 0x7ff99c0613a0 2021-03-09T10:02:34.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800907c0 con 0x7ff980083100 2021-03-09T10:02:34.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:34.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:34.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:34.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:34.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.409+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:34.417 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:34.409+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9983 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:34.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.413+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:34.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.413+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:34.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.413+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:34.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.413+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:34.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.413+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980057a60 con 0x7ff980083100 2021-03-09T10:02:34.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.413+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002bad0 con 0x7ff980083100 2021-03-09T10:02:34.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.413+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7ff980024510 tx=0x7ff98c0a2d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:34.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980083100 2021-03-09T10:02:34.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d040 con 0x7ff980083100 2021-03-09T10:02:34.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff980083100 2021-03-09T10:02:34.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.421+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:34.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.421+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:34.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:34.421+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:39.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:39.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:39.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7ff980024510 tx=0x7ff98c0a2d50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:39.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7ff980024510 tx=0x7ff98c0a2d50).stop 2021-03-09T10:02:39.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:39.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:39.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:39.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:39.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:39.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014190 con 0x7ff99c0adc20 2021-03-09T10:02:39.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800562e0 con 0x7ff99c0613a0 2021-03-09T10:02:39.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e7c0 con 0x7ff980051540 2021-03-09T10:02:39.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:39.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:39.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:39.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:39.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.409+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:39.420 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:39.413+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10021 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:39.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.413+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:39.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.413+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:39.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.413+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:39.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.413+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:39.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.413+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003d520 con 0x7ff980051540 2021-03-09T10:02:39.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.413+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009a70 con 0x7ff980051540 2021-03-09T10:02:39.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.413+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7ff9840a5e40 tx=0x7ff9840813d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:39.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98400b4b0 con 0x7ff980051540 2021-03-09T10:02:39.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400b4b0 con 0x7ff980051540 2021-03-09T10:02:39.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840447a0 con 0x7ff980051540 2021-03-09T10:02:39.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.421+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:39.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.421+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:39.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:39.421+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:44.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:44.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:44.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7ff9840a5e40 tx=0x7ff9840813d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:44.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7ff9840a5e40 tx=0x7ff9840813d0).stop 2021-03-09T10:02:44.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:44.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:44.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:44.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:44.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:44.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800231e0 con 0x7ff99c0adc20 2021-03-09T10:02:44.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800946a0 con 0x7ff99c0613a0 2021-03-09T10:02:44.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038250 con 0x7ff980083100 2021-03-09T10:02:44.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:44.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:44.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:44.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:44.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.413+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:44.421 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:44.413+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10056 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:44.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.417+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:44.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.417+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:44.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.417+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:44.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.417+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:44.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.417+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000a8e0 con 0x7ff980083100 2021-03-09T10:02:44.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.417+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003d520 con 0x7ff980083100 2021-03-09T10:02:44.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.417+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7ff984046260 tx=0x7ff984016710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:44.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98404e4d0 con 0x7ff980083100 2021-03-09T10:02:44.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98404e4d0 con 0x7ff980083100 2021-03-09T10:02:44.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.425+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98402cb20 con 0x7ff980083100 2021-03-09T10:02:44.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.425+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:44.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.425+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:44.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:44.425+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:49.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:49.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:49.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7ff984046260 tx=0x7ff984016710).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:49.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7ff984046260 tx=0x7ff984016710).stop 2021-03-09T10:02:49.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:49.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027440 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:49.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:49.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:49.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:49.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038250 con 0x7ff99c0adc20 2021-03-09T10:02:49.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055c70 con 0x7ff99c0613a0 2021-03-09T10:02:49.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce00 con 0x7ff980051540 2021-03-09T10:02:49.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:49.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:49.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.413+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:49.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.417+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:49.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:49.423 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:49.417+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10094 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:49.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:49.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.421+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:49.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:49.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.421+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:49.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003d520 con 0x7ff980051540 2021-03-09T10:02:49.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800199d0 con 0x7ff980051540 2021-03-09T10:02:49.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.421+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7ff98003dd50 tx=0x7ff994007710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:49.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99401ce40 con 0x7ff980051540 2021-03-09T10:02:49.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.425+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99401ce40 con 0x7ff980051540 2021-03-09T10:02:49.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.425+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:02:49.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.425+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:49.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.429+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:49.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:49.429+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:54.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:54.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:54.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7ff98003dd50 tx=0x7ff994007710).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:54.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7ff98003dd50 tx=0x7ff994007710).stop 2021-03-09T10:02:54.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:54.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026f50 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:54.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:54.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:54.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:54.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004e340 con 0x7ff99c0adc20 2021-03-09T10:02:54.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038250 con 0x7ff99c0613a0 2021-03-09T10:02:54.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800432f0 con 0x7ff980083100 2021-03-09T10:02:54.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:54.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:54.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:54.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:54.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:54.427 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:54.417+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10127 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:54.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:54.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:54.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.417+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:54.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.421+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:54.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.421+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800231e0 con 0x7ff980083100 2021-03-09T10:02:54.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.421+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000a8e0 con 0x7ff980083100 2021-03-09T10:02:54.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.421+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 rx=0x7ff9840035a0 tx=0x7ff98402af60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:54.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98404e4d0 con 0x7ff980083100 2021-03-09T10:02:54.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98402de20 con 0x7ff980083100 2021-03-09T10:02:54.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.425+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840447a0 con 0x7ff980083100 2021-03-09T10:02:54.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.425+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:54.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.425+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:54.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:54.425+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:02:58.001 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:02:58.015 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:02:59.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:59.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:59.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 rx=0x7ff9840035a0 tx=0x7ff98402af60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:59.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 rx=0x7ff9840035a0 tx=0x7ff98402af60).stop 2021-03-09T10:02:59.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:59.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=CLOSED pgs=3093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:59.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:59.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:59.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:59.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008b4b0 con 0x7ff99c0adc20 2021-03-09T10:02:59.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce00 con 0x7ff99c0613a0 2021-03-09T10:02:59.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038250 con 0x7ff980051540 2021-03-09T10:02:59.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:02:59.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:02:59.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.416+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:59.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.420+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:59.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:59.430 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:02:59.420+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10164 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:02:59.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:59.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:59.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:59.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:59.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094840 con 0x7ff980051540 2021-03-09T10:02:59.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003d520 con 0x7ff980051540 2021-03-09T10:02:59.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.424+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff99401ce40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:59.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980051540 2021-03-09T10:02:59.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.428+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980051540 2021-03-09T10:02:59.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:02:59.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.432+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:59.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.432+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:59.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:02:59.432+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:04.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980027660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:04.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980027660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:04.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff99401ce40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:04.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff99401ce40).stop 2021-03-09T10:03:04.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980027660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:04.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 unknown :-1 s=CLOSED pgs=3103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:04.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:04.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:04.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:04.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008b4b0 con 0x7ff99c0adc20 2021-03-09T10:03:04.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800432f0 con 0x7ff99c0613a0 2021-03-09T10:03:04.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:04.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003c80 con 0x7ff980083100 2021-03-09T10:03:04.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:04.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:04.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:04.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:04.431 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:04.420+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10192 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:04.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:04.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:04.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:04.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:04.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003d520 con 0x7ff980083100 2021-03-09T10:03:04.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800231e0 con 0x7ff980083100 2021-03-09T10:03:04.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840167c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:04.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.428+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984006aa0 con 0x7ff980083100 2021-03-09T10:03:04.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.428+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984003900 con 0x7ff980083100 2021-03-09T10:03:04.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840169f0 con 0x7ff980083100 2021-03-09T10:03:04.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.432+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:04.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.432+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:04.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:04.432+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:09.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:09.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:09.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840167c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:09.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff9840167c0).stop 2021-03-09T10:03:09.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:09.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:09.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:09.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:09.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:09.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d9b0 con 0x7ff99c0adc20 2021-03-09T10:03:09.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038250 con 0x7ff99c0613a0 2021-03-09T10:03:09.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce00 con 0x7ff980051540 2021-03-09T10:03:09.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:09.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:09.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:09.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:09.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:09.434 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:09.424+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10247 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:09.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:09.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:09.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:09.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:09.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055c70 con 0x7ff980051540 2021-03-09T10:03:09.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003d520 con 0x7ff980051540 2021-03-09T10:03:09.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.428+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002e290 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff984028f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:09.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.428+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984035a10 con 0x7ff980051540 2021-03-09T10:03:09.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.428+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984035b70 con 0x7ff980051540 2021-03-09T10:03:09.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984035a10 con 0x7ff980051540 2021-03-09T10:03:09.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.432+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:09.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.432+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:09.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:09.432+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:14.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:14.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:14.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002e290 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff984028f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:14.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002e290 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 rx=0x7ff98407b560 tx=0x7ff984028f70).stop 2021-03-09T10:03:14.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002e290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:14.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002e290 unknown :-1 s=CLOSED pgs=3125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:14.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:14.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:14.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:14.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:14.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038250 con 0x7ff99c0adc20 2021-03-09T10:03:14.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:14.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800432f0 con 0x7ff99c0613a0 2021-03-09T10:03:14.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce00 con 0x7ff980083100 2021-03-09T10:03:14.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:14.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:14.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.424+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:14.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:14.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:14.438 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:14.428+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10270 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:14.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:14.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:14.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009a70 con 0x7ff980083100 2021-03-09T10:03:14.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094840 con 0x7ff980083100 2021-03-09T10:03:14.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940114e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:14.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994066880 con 0x7ff980083100 2021-03-09T10:03:14.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980083100 2021-03-09T10:03:14.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994066d80 con 0x7ff980083100 2021-03-09T10:03:14.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.436+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:14.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.436+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:14.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:14.436+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:19.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:19.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:19.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940114e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:19.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940114e0).stop 2021-03-09T10:03:19.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:19.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:19.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:19.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:19.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980017a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:19.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093110 con 0x7ff99c0adc20 2021-03-09T10:03:19.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018e40 con 0x7ff99c0613a0 2021-03-09T10:03:19.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019690 con 0x7ff980051540 2021-03-09T10:03:19.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:19.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:19.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:19.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:19.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980017a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:19.440 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:19.428+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10309 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:19.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.432+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:19.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.432+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:19.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.432+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:19.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.432+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:19.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.432+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800199d0 con 0x7ff980051540 2021-03-09T10:03:19.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.432+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009a70 con 0x7ff980051540 2021-03-09T10:03:19.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.432+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980017a00 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 rx=0x7ff9940aafd0 tx=0x7ff994078220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:19.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069090 con 0x7ff980051540 2021-03-09T10:03:19.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081af0 con 0x7ff980051540 2021-03-09T10:03:19.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:03:19.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.440+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:19.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.440+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:19.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:19.440+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:24.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980017a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:03:24.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980017a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:24.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980017a00 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 rx=0x7ff9940aafd0 tx=0x7ff994078220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:24.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980017a00 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 rx=0x7ff9940aafd0 tx=0x7ff994078220).stop 2021-03-09T10:03:24.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980017a00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:24.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980017a00 unknown :-1 s=CLOSED pgs=3146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:24.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800903e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:24.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:24.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:24.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038250 con 0x7ff99c0adc20 2021-03-09T10:03:24.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:24.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:24.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:24.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:24.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce00 con 0x7ff99c0613a0 2021-03-09T10:03:24.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019690 con 0x7ff980083100 2021-03-09T10:03:24.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800903e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:24.441 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:24.432+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10350 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:24.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:24.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:24.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:24.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:24.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000a8e0 con 0x7ff980083100 2021-03-09T10:03:24.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055c70 con 0x7ff980083100 2021-03-09T10:03:24.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.432+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800903e0 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 rx=0x7ff98407a740 tx=0x7ff98407a600).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:24.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98403ac00 con 0x7ff980083100 2021-03-09T10:03:24.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98403a9d0 con 0x7ff980083100 2021-03-09T10:03:24.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840813b0 con 0x7ff980083100 2021-03-09T10:03:24.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.440+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:24.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.440+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:24.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:24.440+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:28.064 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:03:28.067 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:03:29.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800903e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:29.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800903e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:29.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800903e0 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 rx=0x7ff98407a740 tx=0x7ff98407a600).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:29.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800903e0 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 rx=0x7ff98407a740 tx=0x7ff98407a600).stop 2021-03-09T10:03:29.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800903e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:29.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800903e0 unknown :-1 s=CLOSED pgs=3156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:29.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:29.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:29.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:29.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038250 con 0x7ff99c0adc20 2021-03-09T10:03:29.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800090c0 con 0x7ff99c0613a0 2021-03-09T10:03:29.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019690 con 0x7ff980051540 2021-03-09T10:03:29.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:29.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:29.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:29.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.432+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:29.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.436+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:29.443 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:29.436+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10383 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:29.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.436+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:29.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.436+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:29.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.436+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:29.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.436+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:29.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.436+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980023380 con 0x7ff980051540 2021-03-09T10:03:29.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.436+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980057a60 con 0x7ff980051540 2021-03-09T10:03:29.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.440+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 rx=0x7ff980012450 tx=0x7ff98c0066c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:29.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T10:03:29.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980051540 2021-03-09T10:03:29.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.448+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980051540 2021-03-09T10:03:29.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.448+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:29.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.448+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:29.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:29.448+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:34.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:34.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:34.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 rx=0x7ff980012450 tx=0x7ff98c0066c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:34.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 rx=0x7ff980012450 tx=0x7ff98c0066c0).stop 2021-03-09T10:03:34.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98004aa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:34.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:34.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:34.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:34.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:34.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800092d0 con 0x7ff99c0adc20 2021-03-09T10:03:34.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800211e0 con 0x7ff99c0613a0 2021-03-09T10:03:34.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e550 con 0x7ff980083100 2021-03-09T10:03:34.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:34.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:34.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:34.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:34.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:34.446 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:34.436+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10422 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:34.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:34.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:34.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:34.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:34.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009a70 con 0x7ff980083100 2021-03-09T10:03:34.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.436+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000a8e0 con 0x7ff980083100 2021-03-09T10:03:34.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.440+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027660 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98401eab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:34.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984006aa0 con 0x7ff980083100 2021-03-09T10:03:34.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840813b0 con 0x7ff980083100 2021-03-09T10:03:34.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400f180 con 0x7ff980083100 2021-03-09T10:03:34.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.444+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:34.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.448+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:34.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:34.448+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:39.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.436+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:39.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.436+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:39.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.436+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027660 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98401eab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:39.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.436+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027660 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98401eab0).stop 2021-03-09T10:03:39.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:39.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.436+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027660 unknown :-1 s=CLOSED pgs=3177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:39.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.436+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:39.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.436+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:39.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:39.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043490 con 0x7ff99c0adc20 2021-03-09T10:03:39.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c7f0 con 0x7ff99c0613a0 2021-03-09T10:03:39.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ad90 con 0x7ff980051540 2021-03-09T10:03:39.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:39.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:39.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:39.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:39.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:39.449 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:39.440+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10463 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:39.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:39.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:39.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:39.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:39.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009070 con 0x7ff980051540 2021-03-09T10:03:39.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009a70 con 0x7ff980051540 2021-03-09T10:03:39.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.440+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984025f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:39.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98404efc0 con 0x7ff980051540 2021-03-09T10:03:39.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98404f120 con 0x7ff980051540 2021-03-09T10:03:39.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.448+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984016940 con 0x7ff980051540 2021-03-09T10:03:39.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.448+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:39.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.448+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:39.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:39.448+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:44.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:03:44.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:44.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984025f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:44.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 rx=0x7ff98401de50 tx=0x7ff984025f70).stop 2021-03-09T10:03:44.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98004aa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:44.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004aa30 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:44.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:44.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:44.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:44.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:44.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043490 con 0x7ff99c0adc20 2021-03-09T10:03:44.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:44.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008b820 con 0x7ff99c0613a0 2021-03-09T10:03:44.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c7f0 con 0x7ff980083100 2021-03-09T10:03:44.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:44.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:44.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.440+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:44.451 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:44.440+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10490 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:44.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.444+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:44.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.444+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:44.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.444+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:44.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.444+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:44.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.444+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009280 con 0x7ff980083100 2021-03-09T10:03:44.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.444+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009070 con 0x7ff980083100 2021-03-09T10:03:44.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.444+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027800 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 rx=0x7ff984046810 tx=0x7ff984056e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:44.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.448+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984046650 con 0x7ff980083100 2021-03-09T10:03:44.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984046650 con 0x7ff980083100 2021-03-09T10:03:44.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.456+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984046d80 con 0x7ff980083100 2021-03-09T10:03:44.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.456+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:44.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.456+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:44.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:44.456+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:49.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:49.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:49.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027800 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 rx=0x7ff984046810 tx=0x7ff984056e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:49.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027800 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 rx=0x7ff984046810 tx=0x7ff984056e70).stop 2021-03-09T10:03:49.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980027800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:49.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980027800 unknown :-1 s=CLOSED pgs=3200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:49.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:49.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:49.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:49.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce00 con 0x7ff99c0adc20 2021-03-09T10:03:49.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005bb0 con 0x7ff99c0613a0 2021-03-09T10:03:49.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003c30 con 0x7ff980051540 2021-03-09T10:03:49.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:49.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:49.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:49.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:49.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:49.454 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:49.444+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10527 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:49.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:49.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:49.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:49.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:49.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009070 con 0x7ff980051540 2021-03-09T10:03:49.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.444+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980023380 con 0x7ff980051540 2021-03-09T10:03:49.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.448+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000e9d0 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d20 tx=0x7ff98c006650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:49.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075080 con 0x7ff980051540 2021-03-09T10:03:49.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980051540 2021-03-09T10:03:49.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.456+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff980051540 2021-03-09T10:03:49.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.456+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:49.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.456+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:49.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:49.456+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:54.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:54.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:54.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000e9d0 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d20 tx=0x7ff98c006650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:54.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000e9d0 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d20 tx=0x7ff98c006650).stop 2021-03-09T10:03:54.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000e9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:54.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000e9d0 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:54.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:54.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:54.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:54.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049330 con 0x7ff99c0adc20 2021-03-09T10:03:54.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005bb0 con 0x7ff99c0613a0 2021-03-09T10:03:54.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:54.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008a930 con 0x7ff980083100 2021-03-09T10:03:54.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:54.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:54.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.444+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:54.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.448+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:54.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.448+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:54.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.448+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:54.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.448+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:54.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.448+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:54.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.448+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000a8e0 con 0x7ff980083100 2021-03-09T10:03:54.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.448+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009070 con 0x7ff980083100 2021-03-09T10:03:54.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.448+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 rx=0x7ff98c004150 tx=0x7ff98c014af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:54.459 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:54.448+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10566 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:54.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980083100 2021-03-09T10:03:54.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980083100 2021-03-09T10:03:54.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff980083100 2021-03-09T10:03:54.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.452+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:54.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.456+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:54.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:54.456+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:03:58.110 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:03:58.121 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:03:59.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:59.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:59.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 rx=0x7ff98c004150 tx=0x7ff98c014af0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:59.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 rx=0x7ff98c004150 tx=0x7ff98c014af0).stop 2021-03-09T10:03:59.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98004aa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:59.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98004aa30 unknown :-1 s=CLOSED pgs=3221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:59.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:59.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:59.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:59.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088e30 con 0x7ff99c0adc20 2021-03-09T10:03:59.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055030 con 0x7ff99c0613a0 2021-03-09T10:03:59.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c000 con 0x7ff980051540 2021-03-09T10:03:59.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:03:59.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:59.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:03:59.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:59.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.448+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:59.460 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:03:59.452+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10597 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:03:59.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.452+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:59.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.452+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:59.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.452+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:59.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.452+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:59.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.452+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055c40 con 0x7ff980051540 2021-03-09T10:03:59.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.452+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800199d0 con 0x7ff980051540 2021-03-09T10:03:59.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.452+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99400bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:59.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405dce0 con 0x7ff980051540 2021-03-09T10:03:59.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405dce0 con 0x7ff980051540 2021-03-09T10:03:59.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.456+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:03:59.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.456+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:59.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.456+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:59.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:03:59.456+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:04.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980027660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:04.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980027660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:04.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99400bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:04.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99400bfd0).stop 2021-03-09T10:04:04.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980027660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:04.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980027660 unknown :-1 s=CLOSED pgs=3232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:04.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:04.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:04.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:04.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800060d0 con 0x7ff99c0adc20 2021-03-09T10:04:04.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800551d0 con 0x7ff99c0613a0 2021-03-09T10:04:04.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800383f0 con 0x7ff980083100 2021-03-09T10:04:04.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:04.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:04.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:04.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:04.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:04.462 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:04.452+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10636 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:04.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:04.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:04.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:04.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:04.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009a70 con 0x7ff980083100 2021-03-09T10:04:04.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.452+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009280 con 0x7ff980083100 2021-03-09T10:04:04.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.456+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800055d0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 rx=0x7ff984043760 tx=0x7ff98408a470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:04.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.460+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98400f180 con 0x7ff980083100 2021-03-09T10:04:04.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.460+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984043420 con 0x7ff980083100 2021-03-09T10:04:04.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.460+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98402d8b0 con 0x7ff980083100 2021-03-09T10:04:04.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.460+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:04.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.460+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:04.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:04.460+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:09.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:09.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:09.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800055d0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 rx=0x7ff984043760 tx=0x7ff98408a470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:09.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800055d0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 rx=0x7ff984043760 tx=0x7ff98408a470).stop 2021-03-09T10:04:09.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800055d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:09.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800055d0 unknown :-1 s=CLOSED pgs=3242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:09.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:09.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:09.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:09.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:09.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f080 con 0x7ff99c0adc20 2021-03-09T10:04:09.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:09.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050380 con 0x7ff99c0613a0 2021-03-09T10:04:09.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.452+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800551d0 con 0x7ff980051540 2021-03-09T10:04:09.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.456+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:09.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.456+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:09.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.456+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:09.469 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:09.456+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10679 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:09.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.456+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:09.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.456+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:09.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.456+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:09.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.456+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:09.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.456+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009070 con 0x7ff980051540 2021-03-09T10:04:09.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.456+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000a8e0 con 0x7ff980051540 2021-03-09T10:04:09.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.456+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 rx=0x7ff98c00c010 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:09.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.460+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075020 con 0x7ff980051540 2021-03-09T10:04:09.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.460+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d080 con 0x7ff980051540 2021-03-09T10:04:09.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.464+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff980051540 2021-03-09T10:04:09.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.464+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:09.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.464+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:09.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:09.464+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:14.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800275f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:14.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800275f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:14.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 rx=0x7ff98c00c010 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:14.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 rx=0x7ff98c00c010 tx=0x7ff98c003e80).stop 2021-03-09T10:04:14.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800275f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:14.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 unknown :-1 s=CLOSED pgs=3252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:14.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:14.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:14.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:14.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a930 con 0x7ff99c0adc20 2021-03-09T10:04:14.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800383f0 con 0x7ff99c0613a0 2021-03-09T10:04:14.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003edb0 con 0x7ff980083100 2021-03-09T10:04:14.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:14.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:14.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:14.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:14.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.456+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:14.467 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:14.460+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10714 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:14.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.460+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:14.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.460+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:14.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.460+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:14.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.460+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:14.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.460+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980023380 con 0x7ff980083100 2021-03-09T10:04:14.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.460+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055c40 con 0x7ff980083100 2021-03-09T10:04:14.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.460+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d000 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 rx=0x7ff99400b080 tx=0x7ff9940073c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:14.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.460+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980083100 2021-03-09T10:04:14.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.460+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940663c0 con 0x7ff980083100 2021-03-09T10:04:14.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.464+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:04:14.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.464+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:14.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.468+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:14.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:14.468+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:19.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:04:19.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:19.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d000 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 rx=0x7ff99400b080 tx=0x7ff9940073c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:19.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d000 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 rx=0x7ff99400b080 tx=0x7ff9940073c0).stop 2021-03-09T10:04:19.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008d000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:19.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008d000 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:19.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:19.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:19.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:19.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a930 con 0x7ff99c0adc20 2021-03-09T10:04:19.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013a00 con 0x7ff99c0613a0 2021-03-09T10:04:19.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018e10 con 0x7ff980051540 2021-03-09T10:04:19.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:19.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:19.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:19.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:19.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:19.469 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:19.459+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10752 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:19.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:19.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:19.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:19.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:19.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980057a60 con 0x7ff980051540 2021-03-09T10:04:19.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980023380 con 0x7ff980051540 2021-03-09T10:04:19.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.463+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7ff980012870 tx=0x7ff994035070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:19.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980051540 2021-03-09T10:04:19.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940668d0 con 0x7ff980051540 2021-03-09T10:04:19.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994081af0 con 0x7ff980051540 2021-03-09T10:04:19.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.467+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:19.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.467+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:19.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:19.467+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:24.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800275f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:24.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800275f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:24.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7ff980012870 tx=0x7ff994035070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:24.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7ff980012870 tx=0x7ff994035070).stop 2021-03-09T10:04:24.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800275f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:24.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800275f0 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:24.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:24.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:24.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980094840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:24.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dbe0 con 0x7ff99c0adc20 2021-03-09T10:04:24.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980024c50 con 0x7ff99c0613a0 2021-03-09T10:04:24.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b190 con 0x7ff980083100 2021-03-09T10:04:24.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:24.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:24.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:24.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.459+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:24.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.463+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980094840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:24.471 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:24.463+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10778 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:24.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.463+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:24.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.463+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:24.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.463+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:24.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.463+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:24.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.463+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055c40 con 0x7ff980083100 2021-03-09T10:04:24.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.463+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980057a60 con 0x7ff980083100 2021-03-09T10:04:24.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.463+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980094840 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99405d100).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:24.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980083100 2021-03-09T10:04:24.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994093a50 con 0x7ff980083100 2021-03-09T10:04:24.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940668d0 con 0x7ff980083100 2021-03-09T10:04:24.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.471+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:24.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.471+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:24.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:24.471+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:28.254 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:04:28.257 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:04:29.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980094840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:29.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980094840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:29.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980094840 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99405d100).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:29.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980094840 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99405d100).stop 2021-03-09T10:04:29.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980094840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:29.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980094840 unknown :-1 s=CLOSED pgs=3284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:29.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:29.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:29.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:29.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980024c90 con 0x7ff99c0adc20 2021-03-09T10:04:29.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056350 con 0x7ff99c0613a0 2021-03-09T10:04:29.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800275f0 con 0x7ff980051540 2021-03-09T10:04:29.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:29.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:29.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:29.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:29.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.463+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:29.474 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:29.463+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10813 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:29.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.467+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:29.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.467+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:29.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.467+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:29.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.467+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:29.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.467+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003d060 con 0x7ff980051540 2021-03-09T10:04:29.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.467+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009a70 con 0x7ff980051540 2021-03-09T10:04:29.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.467+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088760 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff9840876e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:29.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984016940 con 0x7ff980051540 2021-03-09T10:04:29.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984016aa0 con 0x7ff980051540 2021-03-09T10:04:29.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984016940 con 0x7ff980051540 2021-03-09T10:04:29.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.471+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:29.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.471+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:29.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:29.471+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:34.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:34.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:34.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088760 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff9840876e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:34.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088760 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 rx=0x7ff98407ba20 tx=0x7ff9840876e0).stop 2021-03-09T10:04:34.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:34.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088760 unknown :-1 s=CLOSED pgs=3295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:34.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:34.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800275f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:34.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:34.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051cb0 con 0x7ff99c0adc20 2021-03-09T10:04:34.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055000 con 0x7ff99c0613a0 2021-03-09T10:04:34.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009a70 con 0x7ff980083100 2021-03-09T10:04:34.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:34.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:34.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:34.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:34.476 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:34.467+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10858 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:34.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.467+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800275f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:34.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.471+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:34.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.471+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:34.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.471+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:34.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.471+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:34.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.471+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027e20 con 0x7ff980083100 2021-03-09T10:04:34.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.471+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009070 con 0x7ff980083100 2021-03-09T10:04:34.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.471+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800275f0 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 rx=0x7ff98c084440 tx=0x7ff98c0a9120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:34.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980083100 2021-03-09T10:04:34.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980083100 2021-03-09T10:04:34.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c008d50 con 0x7ff980083100 2021-03-09T10:04:34.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.479+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:34.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.479+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:34.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:34.479+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:39.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800275f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:04:39.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800275f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:39.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800275f0 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 rx=0x7ff98c084440 tx=0x7ff98c0a9120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:39.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800275f0 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 rx=0x7ff98c084440 tx=0x7ff98c0a9120).stop 2021-03-09T10:04:39.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800275f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:39.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800275f0 unknown :-1 s=CLOSED pgs=3310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:39.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:39.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800930d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:39.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:39.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088410 con 0x7ff99c0adc20 2021-03-09T10:04:39.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800139e0 con 0x7ff99c0613a0 2021-03-09T10:04:39.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800886c0 con 0x7ff980051540 2021-03-09T10:04:39.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:39.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:39.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:39.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.467+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:39.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.471+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800930d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:39.479 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:39.471+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10885 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:39.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.475+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:39.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.475+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:39.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.475+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:39.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.475+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:39.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.475+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004ab30 con 0x7ff980051540 2021-03-09T10:04:39.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.475+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027e20 con 0x7ff980051540 2021-03-09T10:04:39.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.475+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800930d0 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 rx=0x7ff98c0a6fd0 tx=0x7ff98c03d1b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:39.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T10:04:39.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0160c0 con 0x7ff980051540 2021-03-09T10:04:39.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T10:04:39.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.479+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:39.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.479+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:39.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:39.479+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:44.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800930d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:44.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800930d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:44.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800930d0 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 rx=0x7ff98c0a6fd0 tx=0x7ff98c03d1b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:44.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800930d0 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 rx=0x7ff98c0a6fd0 tx=0x7ff98c03d1b0).stop 2021-03-09T10:04:44.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800930d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:44.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800930d0 unknown :-1 s=CLOSED pgs=3323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:44.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:44.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:44.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:44.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088410 con 0x7ff99c0adc20 2021-03-09T10:04:44.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025190 con 0x7ff99c0613a0 2021-03-09T10:04:44.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008b840 con 0x7ff980083100 2021-03-09T10:04:44.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:44.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:44.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:44.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:44.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.471+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:44.483 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:44.471+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10931 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:44.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:44.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:44.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:44.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:44.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003edb0 con 0x7ff980083100 2021-03-09T10:04:44.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055c40 con 0x7ff980083100 2021-03-09T10:04:44.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093740 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 rx=0x7ff98001bae0 tx=0x7ff99400bf50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:44.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff980083100 2021-03-09T10:04:44.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99401cdd0 con 0x7ff980083100 2021-03-09T10:04:44.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:04:44.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.479+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:44.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.483+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:44.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:44.483+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:49.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:49.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:49.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093740 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 rx=0x7ff98001bae0 tx=0x7ff99400bf50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:49.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093740 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 rx=0x7ff98001bae0 tx=0x7ff99400bf50).stop 2021-03-09T10:04:49.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:49.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093740 unknown :-1 s=CLOSED pgs=3330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:49.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:49.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:49.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:49.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025190 con 0x7ff99c0adc20 2021-03-09T10:04:49.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088410 con 0x7ff99c0613a0 2021-03-09T10:04:49.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055c40 con 0x7ff980051540 2021-03-09T10:04:49.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:49.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:49.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:49.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:49.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:49.485 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:49.475+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10959 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:49.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:49.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:49.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:49.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:49.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800383f0 con 0x7ff980051540 2021-03-09T10:04:49.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.475+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003d060 con 0x7ff980051540 2021-03-09T10:04:49.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.479+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088620 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984016a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:49.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.483+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840447a0 con 0x7ff980051540 2021-03-09T10:04:49.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840447a0 con 0x7ff980051540 2021-03-09T10:04:49.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984044910 con 0x7ff980051540 2021-03-09T10:04:49.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.487+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:49.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.487+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:49.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:49.487+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:54.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:54.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:54.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088620 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984016a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:54.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088620 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984016a30).stop 2021-03-09T10:04:54.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:54.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088620 unknown :-1 s=CLOSED pgs=3338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:54.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:54.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:54.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:54.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088410 con 0x7ff99c0adc20 2021-03-09T10:04:54.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b580 con 0x7ff99c0613a0 2021-03-09T10:04:54.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980024300 con 0x7ff980083100 2021-03-09T10:04:54.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:54.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:54.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:54.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.479+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:54.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.479+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:54.488 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:54.479+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11002 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:54.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.479+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:54.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.479+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:54.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.479+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:54.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.479+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:54.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.479+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980024830 con 0x7ff980083100 2021-03-09T10:04:54.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.479+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800383f0 con 0x7ff980083100 2021-03-09T10:04:54.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.483+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cf50 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff98402aec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:54.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.483+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98403ad80 con 0x7ff980083100 2021-03-09T10:04:54.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.483+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98403aba0 con 0x7ff980083100 2021-03-09T10:04:54.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98402cb20 con 0x7ff980083100 2021-03-09T10:04:54.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.487+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:54.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.491+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:54.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:54.491+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:04:58.562 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:04:58.565 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:04:59.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:59.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:59.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cf50 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff98402aec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:59.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cf50 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff98402aec0).stop 2021-03-09T10:04:59.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008cf50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:59.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cf50 unknown :-1 s=CLOSED pgs=3350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:59.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:59.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:59.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:59.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055c90 con 0x7ff99c0adc20 2021-03-09T10:04:59.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088410 con 0x7ff99c0613a0 2021-03-09T10:04:59.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013220 con 0x7ff980051540 2021-03-09T10:04:59.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:04:59.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:59.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:04:59.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:59.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.479+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:59.491 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:04:59.483+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11039 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:04:59.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.483+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:59.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.483+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:59.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.483+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:59.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.483+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:59.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.483+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800242b0 con 0x7ff980051540 2021-03-09T10:04:59.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.483+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980024830 con 0x7ff980051540 2021-03-09T10:04:59.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.483+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984059ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:59.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98401e000 con 0x7ff980051540 2021-03-09T10:04:59.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98401e000 con 0x7ff980051540 2021-03-09T10:04:59.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984044020 con 0x7ff980051540 2021-03-09T10:04:59.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.491+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:59.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.491+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:59.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:04:59.491+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:04.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:04.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:04.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984059ae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:04.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984059ae0).stop 2021-03-09T10:05:04.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:04.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 unknown :-1 s=CLOSED pgs=3359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:04.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:04.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:04.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:04.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:04.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:04.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025190 con 0x7ff99c0adc20 2021-03-09T10:05:04.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008acf0 con 0x7ff99c0613a0 2021-03-09T10:05:04.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f740 con 0x7ff980083100 2021-03-09T10:05:04.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:04.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:04.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:04.493 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:04.483+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11075 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:04.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:04.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:04.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:04.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:04.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980048710 con 0x7ff980083100 2021-03-09T10:05:04.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.483+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004ab30 con 0x7ff980083100 2021-03-09T10:05:04.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.487+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 rx=0x7ff98c02a040 tx=0x7ff98c0223b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:04.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:05:04.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:05:04.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980083100 2021-03-09T10:05:04.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.495+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:04.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.495+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:04.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:04.495+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:09.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980024ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:09.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980024ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:09.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 rx=0x7ff98c02a040 tx=0x7ff98c0223b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:09.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 rx=0x7ff98c02a040 tx=0x7ff98c0223b0).stop 2021-03-09T10:05:09.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980024ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:09.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024ab0 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:09.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:09.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:09.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:09.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025190 con 0x7ff99c0adc20 2021-03-09T10:05:09.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013d10 con 0x7ff99c0613a0 2021-03-09T10:05:09.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ab30 con 0x7ff980051540 2021-03-09T10:05:09.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:09.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:09.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:09.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:09.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:09.495 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:09.487+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11109 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:09.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:09.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:09.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:09.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:09.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009240 con 0x7ff980051540 2021-03-09T10:05:09.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800242b0 con 0x7ff980051540 2021-03-09T10:05:09.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.487+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984027b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:09.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984009070 con 0x7ff980051540 2021-03-09T10:05:09.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984027d70 con 0x7ff980051540 2021-03-09T10:05:09.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980051540 2021-03-09T10:05:09.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.495+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:09.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.495+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:09.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:09.495+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:14.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:14.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:14.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984027b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:14.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984027b90).stop 2021-03-09T10:05:14.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:14.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093710 unknown :-1 s=CLOSED pgs=3380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:14.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:14.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:14.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:14.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ab80 con 0x7ff99c0adc20 2021-03-09T10:05:14.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037fc0 con 0x7ff99c0613a0 2021-03-09T10:05:14.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025190 con 0x7ff980083100 2021-03-09T10:05:14.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:14.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:14.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:14.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:14.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.487+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:14.499 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:14.487+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11145 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:14.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:14.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.491+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:14.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:14.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.491+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:14.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055c40 con 0x7ff980083100 2021-03-09T10:05:14.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980048710 con 0x7ff980083100 2021-03-09T10:05:14.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.495+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037ad0 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7ff980056860 tx=0x7ff98c052130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:14.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a0130 con 0x7ff980083100 2021-03-09T10:05:14.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980083100 2021-03-09T10:05:14.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c099120 con 0x7ff980083100 2021-03-09T10:05:14.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.499+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:14.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.503+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:14.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:14.503+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:19.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:19.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:19.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037ad0 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7ff980056860 tx=0x7ff98c052130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:19.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037ad0 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7ff980056860 tx=0x7ff98c052130).stop 2021-03-09T10:05:19.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:19.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037ad0 unknown :-1 s=CLOSED pgs=3394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:19.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:19.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:19.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:19.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011270 con 0x7ff99c0adc20 2021-03-09T10:05:19.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014060 con 0x7ff99c0613a0 2021-03-09T10:05:19.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037fc0 con 0x7ff980051540 2021-03-09T10:05:19.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:19.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:19.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:19.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:19.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:19.501 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:19.491+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11171 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:19.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:19.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:19.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:19.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:19.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003f740 con 0x7ff980051540 2021-03-09T10:05:19.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003edb0 con 0x7ff980051540 2021-03-09T10:05:19.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.491+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013b70 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 rx=0x7ff9800494f0 tx=0x7ff99407a2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:19.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069090 con 0x7ff980051540 2021-03-09T10:05:19.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405c990 con 0x7ff980051540 2021-03-09T10:05:19.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:05:19.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.499+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:19.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.499+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:19.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:19.499+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:24.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:24.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:24.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013b70 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 rx=0x7ff9800494f0 tx=0x7ff99407a2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:24.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013b70 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 rx=0x7ff9800494f0 tx=0x7ff99407a2e0).stop 2021-03-09T10:05:24.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:24.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013b70 unknown :-1 s=CLOSED pgs=3402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:24.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:24.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:24.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:24.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017c10 con 0x7ff99c0adc20 2021-03-09T10:05:24.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090430 con 0x7ff99c0613a0 2021-03-09T10:05:24.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048710 con 0x7ff980083100 2021-03-09T10:05:24.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:24.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:24.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:24.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:24.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:24.502 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:24.495+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11212 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:24.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:24.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:24.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:24.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:24.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008acf0 con 0x7ff980083100 2021-03-09T10:05:24.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009240 con 0x7ff980083100 2021-03-09T10:05:24.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.495+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 rx=0x7ff984005370 tx=0x7ff9840262c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:24.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984027f80 con 0x7ff980083100 2021-03-09T10:05:24.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984003900 con 0x7ff980083100 2021-03-09T10:05:24.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980083100 2021-03-09T10:05:24.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.503+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:24.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.503+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:24.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:24.503+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:28.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:05:28.610 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:05:29.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:29.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:29.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 rx=0x7ff984005370 tx=0x7ff9840262c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:29.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 rx=0x7ff984005370 tx=0x7ff9840262c0).stop 2021-03-09T10:05:29.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:29.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 unknown :-1 s=CLOSED pgs=3415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:29.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:29.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:29.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:29.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090430 con 0x7ff99c0adc20 2021-03-09T10:05:29.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048710 con 0x7ff99c0613a0 2021-03-09T10:05:29.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003bb50 con 0x7ff980051540 2021-03-09T10:05:29.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:29.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:29.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:29.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:29.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:29.505 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:29.495+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11241 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:29.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:29.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:29.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:29.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:29.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004ab30 con 0x7ff980051540 2021-03-09T10:05:29.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.495+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055c40 con 0x7ff980051540 2021-03-09T10:05:29.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.499+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 rx=0x7ff98c006760 tx=0x7ff98c017330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:29.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08e080 con 0x7ff980051540 2021-03-09T10:05:29.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02d070 con 0x7ff980051540 2021-03-09T10:05:29.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T10:05:29.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.503+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:29.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.503+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:29.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:29.503+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:34.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:34.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:34.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 rx=0x7ff98c006760 tx=0x7ff98c017330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:34.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 rx=0x7ff98c006760 tx=0x7ff98c017330).stop 2021-03-09T10:05:34.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:34.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 unknown :-1 s=CLOSED pgs=3423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:34.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:34.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:34.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:34.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005890 con 0x7ff99c0adc20 2021-03-09T10:05:34.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037780 con 0x7ff99c0613a0 2021-03-09T10:05:34.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055c40 con 0x7ff980083100 2021-03-09T10:05:34.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:34.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:34.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:34.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:34.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:34.508 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:34.499+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11284 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:34.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:34.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:34.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:34.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:34.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980048710 con 0x7ff980083100 2021-03-09T10:05:34.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008acf0 con 0x7ff980083100 2021-03-09T10:05:34.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.499+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 rx=0x7ff98408b1e0 tx=0x7ff984023e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:34.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984009070 con 0x7ff980083100 2021-03-09T10:05:34.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400f250 con 0x7ff980083100 2021-03-09T10:05:34.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.511+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980083100 2021-03-09T10:05:34.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.511+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:34.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.511+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:34.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:34.511+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:39.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:39.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:39.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 rx=0x7ff98408b1e0 tx=0x7ff984023e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:39.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 rx=0x7ff98408b1e0 tx=0x7ff984023e50).stop 2021-03-09T10:05:39.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980037930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:39.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980037930 unknown :-1 s=CLOSED pgs=3433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:39.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:39.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:39.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:39.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a980 con 0x7ff99c0adc20 2021-03-09T10:05:39.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e610 con 0x7ff99c0613a0 2021-03-09T10:05:39.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800215f0 con 0x7ff980051540 2021-03-09T10:05:39.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:39.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:39.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:39.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:39.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.498+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:39.510 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:39.498+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11314 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:39.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.502+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:39.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.502+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:39.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.502+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:39.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.502+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:39.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.502+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003bb50 con 0x7ff980051540 2021-03-09T10:05:39.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.502+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980048710 con 0x7ff980051540 2021-03-09T10:05:39.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.502+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 rx=0x7ff98404f8f0 tx=0x7ff9840779d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:39.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984040000 con 0x7ff980051540 2021-03-09T10:05:39.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98407a1d0 con 0x7ff980051540 2021-03-09T10:05:39.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.510+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984005000 con 0x7ff980051540 2021-03-09T10:05:39.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.510+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:39.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.510+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:39.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:39.510+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:44.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:44.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:44.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 rx=0x7ff98404f8f0 tx=0x7ff9840779d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:44.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 rx=0x7ff98404f8f0 tx=0x7ff9840779d0).stop 2021-03-09T10:05:44.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:44.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013810 unknown :-1 s=CLOSED pgs=3444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:44.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:44.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:44.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:44.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001ab20 con 0x7ff99c0adc20 2021-03-09T10:05:44.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980024ca0 con 0x7ff99c0613a0 2021-03-09T10:05:44.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e610 con 0x7ff980083100 2021-03-09T10:05:44.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:44.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:44.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:44.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:44.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:44.513 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:44.502+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11349 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:44.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:44.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:44.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:44.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:44.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055df0 con 0x7ff980083100 2021-03-09T10:05:44.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.502+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003bb50 con 0x7ff980083100 2021-03-09T10:05:44.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055900 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff98407b520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:44.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.510+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98402ad00 con 0x7ff980083100 2021-03-09T10:05:44.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.510+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984000ba0 con 0x7ff980083100 2021-03-09T10:05:44.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98404f3e0 con 0x7ff980083100 2021-03-09T10:05:44.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.514+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:44.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.514+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:44.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:44.514+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:49.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980055900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:49.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980055900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:49.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055900 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff98407b520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:49.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055900 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 rx=0x7ff98402b010 tx=0x7ff98407b520).stop 2021-03-09T10:05:49.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980055900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:49.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055900 unknown :-1 s=CLOSED pgs=3455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:49.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:49.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:49.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:49.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001acc0 con 0x7ff99c0adc20 2021-03-09T10:05:49.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e610 con 0x7ff99c0613a0 2021-03-09T10:05:49.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003fbb0 con 0x7ff980051540 2021-03-09T10:05:49.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:49.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:49.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:49.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:49.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:49.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:49.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:49.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:49.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:49.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980028290 con 0x7ff980051540 2021-03-09T10:05:49.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055df0 con 0x7ff980051540 2021-03-09T10:05:49.516 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:49.506+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11392 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:49.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013830 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff984033e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:49.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840a5470 con 0x7ff980051540 2021-03-09T10:05:49.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98402ad00 con 0x7ff980051540 2021-03-09T10:05:49.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.510+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840a5d80 con 0x7ff980051540 2021-03-09T10:05:49.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.510+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:49.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.510+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:49.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:49.510+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:54.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:05:54.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:54.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013830 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff984033e30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:54.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013830 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff984033e30).stop 2021-03-09T10:05:54.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:54.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013830 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:54.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:54.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:54.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800226b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:54.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c190 con 0x7ff99c0adc20 2021-03-09T10:05:54.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800556c0 con 0x7ff99c0613a0 2021-03-09T10:05:54.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055800 con 0x7ff980083100 2021-03-09T10:05:54.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:54.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:54.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:54.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:54.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800226b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:54.518 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:54.506+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11422 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:54.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:54.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:54.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:54.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:54.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003bb50 con 0x7ff980083100 2021-03-09T10:05:54.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.506+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f740 con 0x7ff980083100 2021-03-09T10:05:54.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.510+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800226b0 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff99407a180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:54.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.510+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980083100 2021-03-09T10:05:54.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.510+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994082410 con 0x7ff980083100 2021-03-09T10:05:54.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:05:54.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.514+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:54.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.518+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:54.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:54.518+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:05:58.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:05:58.648 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:05:59.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800226b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:59.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800226b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:59.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800226b0 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff99407a180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:59.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800226b0 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff99407a180).stop 2021-03-09T10:05:59.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800226b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:59.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800226b0 unknown :-1 s=CLOSED pgs=3476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:59.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:59.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:59.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:59.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dc90 con 0x7ff99c0adc20 2021-03-09T10:05:59.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d9b0 con 0x7ff99c0613a0 2021-03-09T10:05:59.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c330 con 0x7ff980051540 2021-03-09T10:05:59.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:05:59.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.506+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:59.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.510+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:05:59.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.510+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:59.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.510+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:59.520 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:05:59.510+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11458 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:05:59.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.510+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:59.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.510+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:59.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.510+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:59.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.510+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:59.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.510+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003f740 con 0x7ff980051540 2021-03-09T10:05:59.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.510+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003bb50 con 0x7ff980051540 2021-03-09T10:05:59.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800556a0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 rx=0x7ff9800448b0 tx=0x7ff994081790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:59.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:05:59.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994078130 con 0x7ff980051540 2021-03-09T10:05:59.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700d0 con 0x7ff980051540 2021-03-09T10:05:59.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.522+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:59.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.522+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:59.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:05:59.522+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:04.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:04.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:04.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800556a0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 rx=0x7ff9800448b0 tx=0x7ff994081790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:04.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800556a0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 rx=0x7ff9800448b0 tx=0x7ff994081790).stop 2021-03-09T10:06:04.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800556a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:04.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800556a0 unknown :-1 s=CLOSED pgs=3493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:04.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:04.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:04.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:04.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c4d0 con 0x7ff99c0adc20 2021-03-09T10:06:04.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ce0 con 0x7ff99c0613a0 2021-03-09T10:06:04.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e7a0 con 0x7ff980083100 2021-03-09T10:06:04.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:04.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:04.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:04.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:04.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.510+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:04.525 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:04.514+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11495 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:04.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.514+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:04.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.514+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:04.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.514+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:04.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.514+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:04.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.514+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003bb50 con 0x7ff980083100 2021-03-09T10:06:04.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.514+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980028290 con 0x7ff980083100 2021-03-09T10:06:04.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.514+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013670 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 rx=0x7ff984035540 tx=0x7ff9840090e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:04.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984095e70 con 0x7ff980083100 2021-03-09T10:06:04.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98407a000 con 0x7ff980083100 2021-03-09T10:06:04.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984093390 con 0x7ff980083100 2021-03-09T10:06:04.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.518+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:04.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.522+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:04.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:04.522+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:09.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:09.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:09.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013670 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 rx=0x7ff984035540 tx=0x7ff9840090e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:09.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013670 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 rx=0x7ff984035540 tx=0x7ff9840090e0).stop 2021-03-09T10:06:09.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:09.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013670 unknown :-1 s=CLOSED pgs=3498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:09.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:09.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:09.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:09.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008b4b0 con 0x7ff99c0adc20 2021-03-09T10:06:09.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095790 con 0x7ff99c0613a0 2021-03-09T10:06:09.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c670 con 0x7ff980051540 2021-03-09T10:06:09.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:09.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:09.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:09.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:09.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:09.526 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:09.514+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11530 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:09.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:09.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:09.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:09.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:09.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003de30 con 0x7ff980051540 2021-03-09T10:06:09.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f740 con 0x7ff980051540 2021-03-09T10:06:09.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.518+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 rx=0x7ff994065ea0 tx=0x7ff99400bed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:09.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:06:09.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99401eef0 con 0x7ff980051540 2021-03-09T10:06:09.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.526+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700e0 con 0x7ff980051540 2021-03-09T10:06:09.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.526+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:09.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.526+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:09.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:09.526+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:14.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800936b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:14.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800936b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:14.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 rx=0x7ff994065ea0 tx=0x7ff99400bed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:14.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 rx=0x7ff994065ea0 tx=0x7ff99400bed0).stop 2021-03-09T10:06:14.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800936b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:14.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 unknown :-1 s=CLOSED pgs=3508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:14.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800134b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:14.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:14.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:14.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090750 con 0x7ff99c0adc20 2021-03-09T10:06:14.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008b4b0 con 0x7ff99c0613a0 2021-03-09T10:06:14.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002c810 con 0x7ff980083100 2021-03-09T10:06:14.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:14.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.514+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:14.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.518+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:14.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.518+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:14.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.518+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800134b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:14.528 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:14.518+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11568 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:14.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.518+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:14.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.518+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:14.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.518+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:14.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.518+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:14.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.518+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008d9b0 con 0x7ff980083100 2021-03-09T10:06:14.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.518+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003bb50 con 0x7ff980083100 2021-03-09T10:06:14.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.522+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800134b0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840480e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:14.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980083100 2021-03-09T10:06:14.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984081470 con 0x7ff980083100 2021-03-09T10:06:14.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.526+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984095cd0 con 0x7ff980083100 2021-03-09T10:06:14.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.526+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:14.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.530+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:14.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:14.530+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:19.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800134b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:19.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800134b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:19.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800134b0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840480e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:19.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800134b0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff9840480e0).stop 2021-03-09T10:06:19.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800134b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:19.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800134b0 unknown :-1 s=CLOSED pgs=3522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:19.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:19.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:19.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:19.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088b50 con 0x7ff99c0adc20 2021-03-09T10:06:19.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037980 con 0x7ff99c0613a0 2021-03-09T10:06:19.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f780 con 0x7ff980051540 2021-03-09T10:06:19.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:19.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:19.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:19.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:19.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:19.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:19.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:19.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:19.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:19.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980028290 con 0x7ff980051540 2021-03-09T10:06:19.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003de30 con 0x7ff980051540 2021-03-09T10:06:19.532 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:19.522+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11612 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:19.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994066070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:19.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980051540 2021-03-09T10:06:19.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:06:19.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.526+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:06:19.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.526+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:19.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.526+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:19.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:19.526+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:24.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800936b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:24.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800936b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:24.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994066070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:24.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994066070).stop 2021-03-09T10:06:24.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800936b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:24.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 unknown :-1 s=CLOSED pgs=3529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:24.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:24.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:24.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:24.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001ae60 con 0x7ff99c0adc20 2021-03-09T10:06:24.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090420 con 0x7ff99c0613a0 2021-03-09T10:06:24.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800084d0 con 0x7ff980083100 2021-03-09T10:06:24.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:24.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:24.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:24.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:24.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.522+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:24.535 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:24.522+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11650 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:24.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.526+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:24.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.526+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:24.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.526+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:24.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.526+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:24.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.526+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003bb50 con 0x7ff980083100 2021-03-09T10:06:24.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.526+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008d9b0 con 0x7ff980083100 2021-03-09T10:06:24.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.526+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055650 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984035500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:24.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.526+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984095cd0 con 0x7ff980083100 2021-03-09T10:06:24.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984095a90 con 0x7ff980083100 2021-03-09T10:06:24.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980083100 2021-03-09T10:06:24.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.534+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:24.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.534+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:24.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:24.534+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:28.772 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:06:28.775 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:06:29.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980055650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:29.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980055650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:29.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055650 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984035500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:29.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055650 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984035500).stop 2021-03-09T10:06:29.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980055650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:29.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980055650 unknown :-1 s=CLOSED pgs=3542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:29.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:29.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:29.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:29.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800084d0 con 0x7ff99c0adc20 2021-03-09T10:06:29.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037940 con 0x7ff99c0613a0 2021-03-09T10:06:29.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008aca0 con 0x7ff980051540 2021-03-09T10:06:29.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:29.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:29.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:29.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:29.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:29.540 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:29.526+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11685 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:29.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:29.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:29.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:29.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:29.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003e310 con 0x7ff980051540 2021-03-09T10:06:29.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003bb50 con 0x7ff980051540 2021-03-09T10:06:29.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.526+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800133c0 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff98405a470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:29.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98404f250 con 0x7ff980051540 2021-03-09T10:06:29.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980051540 2021-03-09T10:06:29.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980051540 2021-03-09T10:06:29.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.534+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:29.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.534+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:29.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:29.534+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:34.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:34.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:34.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800133c0 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff98405a470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:34.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800133c0 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff98405a470).stop 2021-03-09T10:06:34.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800133c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:34.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800133c0 unknown :-1 s=CLOSED pgs=3551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:34.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:34.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:34.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:34.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800084d0 con 0x7ff99c0adc20 2021-03-09T10:06:34.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003bb50 con 0x7ff99c0613a0 2021-03-09T10:06:34.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008aca0 con 0x7ff980083100 2021-03-09T10:06:34.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:34.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:34.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:34.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:34.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:34.540 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:34.530+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11716 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:34.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:34.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:34.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:34.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:34.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055650 con 0x7ff980083100 2021-03-09T10:06:34.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003e310 con 0x7ff980083100 2021-03-09T10:06:34.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.530+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984000c10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:34.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98405bec0 con 0x7ff980083100 2021-03-09T10:06:34.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98405bec0 con 0x7ff980083100 2021-03-09T10:06:34.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984082900 con 0x7ff980083100 2021-03-09T10:06:34.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.542+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:34.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.542+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:34.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:34.542+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:39.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:39.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:39.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984000c10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:39.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff984000c10).stop 2021-03-09T10:06:39.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:39.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=CLOSED pgs=3565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:39.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:39.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:39.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:39.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057a60 con 0x7ff99c0adc20 2021-03-09T10:06:39.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009d90 con 0x7ff99c0613a0 2021-03-09T10:06:39.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008aca0 con 0x7ff980051540 2021-03-09T10:06:39.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:39.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:39.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:39.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:39.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:39.541 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:39.530+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11749 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:39.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:39.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:39.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:39.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:39.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003e310 con 0x7ff980051540 2021-03-09T10:06:39.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.530+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004ab30 con 0x7ff980051540 2021-03-09T10:06:39.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.534+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800557e0 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 rx=0x7ff980032570 tx=0x7ff98c0173a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:39.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075080 con 0x7ff980051540 2021-03-09T10:06:39.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c004070 con 0x7ff980051540 2021-03-09T10:06:39.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:06:39.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.542+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:39.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.542+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:39.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:39.542+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:44.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:44.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:44.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800557e0 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 rx=0x7ff980032570 tx=0x7ff98c0173a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:44.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800557e0 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 rx=0x7ff980032570 tx=0x7ff98c0173a0).stop 2021-03-09T10:06:44.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800557e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:44.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800557e0 unknown :-1 s=CLOSED pgs=3573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:44.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:44.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:44.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:44.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009d90 con 0x7ff99c0adc20 2021-03-09T10:06:44.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980021190 con 0x7ff99c0613a0 2021-03-09T10:06:44.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800936b0 con 0x7ff980083100 2021-03-09T10:06:44.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:44.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:44.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:44.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:44.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:44.543 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:44.534+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11794 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:44.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:44.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:44.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:44.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:44.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003bb50 con 0x7ff980083100 2021-03-09T10:06:44.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.534+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055650 con 0x7ff980083100 2021-03-09T10:06:44.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.538+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 rx=0x7ff984095a60 tx=0x7ff98405b190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:44.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984009070 con 0x7ff980083100 2021-03-09T10:06:44.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984006b70 con 0x7ff980083100 2021-03-09T10:06:44.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.546+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98404f8e0 con 0x7ff980083100 2021-03-09T10:06:44.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.546+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:44.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.546+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:44.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:44.546+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:49.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:49.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:49.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 rx=0x7ff984095a60 tx=0x7ff98405b190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:49.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 rx=0x7ff984095a60 tx=0x7ff98405b190).stop 2021-03-09T10:06:49.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:49.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=CLOSED pgs=3587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:49.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:49.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:49.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:49.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e500 con 0x7ff99c0adc20 2021-03-09T10:06:49.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005a50 con 0x7ff99c0613a0 2021-03-09T10:06:49.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b030 con 0x7ff980051540 2021-03-09T10:06:49.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:49.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:49.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:49.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:49.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:49.547 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:49.538+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11831 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:49.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:49.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:49.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:49.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:49.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005870 con 0x7ff980051540 2021-03-09T10:06:49.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003bb50 con 0x7ff980051540 2021-03-09T10:06:49.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.538+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 rx=0x7ff98400f3d0 tx=0x7ff9840090e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:49.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980051540 2021-03-09T10:06:49.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984082900 con 0x7ff980051540 2021-03-09T10:06:49.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984082900 con 0x7ff980051540 2021-03-09T10:06:49.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.550+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:49.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.550+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:49.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:49.550+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:54.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800936b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:54.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800936b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:54.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 rx=0x7ff98400f3d0 tx=0x7ff9840090e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:54.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 rx=0x7ff98400f3d0 tx=0x7ff9840090e0).stop 2021-03-09T10:06:54.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800936b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:54.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800936b0 unknown :-1 s=CLOSED pgs=3595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:54.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:54.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:54.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:54.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005a50 con 0x7ff99c0adc20 2021-03-09T10:06:54.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009d90 con 0x7ff99c0613a0 2021-03-09T10:06:54.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002e4b0 con 0x7ff980083100 2021-03-09T10:06:54.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:54.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:54.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:54.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:54.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:54.549 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:54.538+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11866 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:54.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:54.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:54.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:54.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:54.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800084f0 con 0x7ff980083100 2021-03-09T10:06:54.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.538+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003e310 con 0x7ff980083100 2021-03-09T10:06:54.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.542+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 rx=0x7ff980056110 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:54.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980083100 2021-03-09T10:06:54.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:06:54.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c046da0 con 0x7ff980083100 2021-03-09T10:06:54.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.550+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:54.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.550+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:54.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:54.550+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:06:58.830 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:06:58.833 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:06:59.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:59.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:59.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 rx=0x7ff980056110 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:59.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 rx=0x7ff980056110 tx=0x7ff98c003e80).stop 2021-03-09T10:06:59.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:59.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=CLOSED pgs=3605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:59.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:59.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:59.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:59.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d160 con 0x7ff99c0adc20 2021-03-09T10:06:59.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005a50 con 0x7ff99c0613a0 2021-03-09T10:06:59.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090240 con 0x7ff980051540 2021-03-09T10:06:59.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:06:59.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:59.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:06:59.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:59.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:59.552 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:06:59.542+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11897 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:06:59.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:59.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:59.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:59.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:59.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003e310 con 0x7ff980051540 2021-03-09T10:06:59.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.542+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800084f0 con 0x7ff980051540 2021-03-09T10:06:59.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.546+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093640 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 rx=0x7ff98003d6d0 tx=0x7ff98c0a2f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:59.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.546+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980051540 2021-03-09T10:06:59.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.546+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079f90 con 0x7ff980051540 2021-03-09T10:06:59.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.554+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c08f9b0 con 0x7ff980051540 2021-03-09T10:06:59.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.554+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:59.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.554+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:59.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:06:59.554+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:04.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:04.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:04.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093640 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 rx=0x7ff98003d6d0 tx=0x7ff98c0a2f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:04.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093640 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 rx=0x7ff98003d6d0 tx=0x7ff98c0a2f00).stop 2021-03-09T10:07:04.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:04.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093640 unknown :-1 s=CLOSED pgs=3617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:04.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:04.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:04.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:04.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d110 con 0x7ff99c0adc20 2021-03-09T10:07:04.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043340 con 0x7ff99c0613a0 2021-03-09T10:07:04.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043480 con 0x7ff980083100 2021-03-09T10:07:04.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:04.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:04.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:04.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:04.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:04.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:04.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:04.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:04.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:04.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054e40 con 0x7ff980083100 2021-03-09T10:07:04.557 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:04.546+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11938 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:04.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005870 con 0x7ff980083100 2021-03-09T10:07:04.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98401e790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:04.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98402f000 con 0x7ff980083100 2021-03-09T10:07:04.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.546+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984000ba0 con 0x7ff980083100 2021-03-09T10:07:04.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.554+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98404f8e0 con 0x7ff980083100 2021-03-09T10:07:04.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.554+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:04.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.554+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:04.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:04.554+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:09.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:09.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:09.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98401e790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:09.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff98401e790).stop 2021-03-09T10:07:09.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:09.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=CLOSED pgs=3625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:09.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:09.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:09.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:09.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009d90 con 0x7ff99c0adc20 2021-03-09T10:07:09.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800093b0 con 0x7ff99c0613a0 2021-03-09T10:07:09.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057a30 con 0x7ff980051540 2021-03-09T10:07:09.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:09.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:09.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:09.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:09.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:09.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:09.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:09.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:09.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:09.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005870 con 0x7ff980051540 2021-03-09T10:07:09.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054e40 con 0x7ff980051540 2021-03-09T10:07:09.559 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:09.550+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11961 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:09.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 rx=0x7ff984049060 tx=0x7ff9840961d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:09.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984000ba0 con 0x7ff980051540 2021-03-09T10:07:09.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98407abc0 con 0x7ff980051540 2021-03-09T10:07:09.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.554+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98401e870 con 0x7ff980051540 2021-03-09T10:07:09.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.554+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:09.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.554+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:09.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:09.554+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:14.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800935f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:14.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800935f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:14.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 rx=0x7ff984049060 tx=0x7ff9840961d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:14.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 rx=0x7ff984049060 tx=0x7ff9840961d0).stop 2021-03-09T10:07:14.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800935f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:14.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 unknown :-1 s=CLOSED pgs=3636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:14.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:14.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:14.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:14.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800093b0 con 0x7ff99c0adc20 2021-03-09T10:07:14.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:14.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003fbb0 con 0x7ff99c0613a0 2021-03-09T10:07:14.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037790 con 0x7ff980083100 2021-03-09T10:07:14.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:14.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:14.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:14.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:14.560 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:14.550+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12003 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:14.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:14.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:14.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:14.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:14.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003bb50 con 0x7ff980083100 2021-03-09T10:07:14.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.550+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980028290 con 0x7ff980083100 2021-03-09T10:07:14.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.554+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800432d0 secure :-1 s=READY pgs=3648 cs=0 l=1 rev1=1 rx=0x7ff994066be0 tx=0x7ff994011470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:14.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.554+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980083100 2021-03-09T10:07:14.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.558+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405c990 con 0x7ff980083100 2021-03-09T10:07:14.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.562+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:07:14.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.562+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:14.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.562+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:14.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:14.562+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:19.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800432d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:07:19.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800432d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:19.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800432d0 secure :-1 s=READY pgs=3648 cs=0 l=1 rev1=1 rx=0x7ff994066be0 tx=0x7ff994011470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:19.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800432d0 secure :-1 s=READY pgs=3648 cs=0 l=1 rev1=1 rx=0x7ff994066be0 tx=0x7ff994011470).stop 2021-03-09T10:07:19.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800432d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:19.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800432d0 unknown :-1 s=CLOSED pgs=3648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:19.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:19.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:19.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800937d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:19.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800557f0 con 0x7ff99c0adc20 2021-03-09T10:07:19.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009d90 con 0x7ff99c0613a0 2021-03-09T10:07:19.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e240 con 0x7ff980051540 2021-03-09T10:07:19.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:19.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:19.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:19.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:19.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800937d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:19.564 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:19.553+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12034 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:19.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:19.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:19.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:19.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:19.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980028290 con 0x7ff980051540 2021-03-09T10:07:19.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003bb50 con 0x7ff980051540 2021-03-09T10:07:19.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.553+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800937d0 secure :-1 s=READY pgs=3657 cs=0 l=1 rev1=1 rx=0x7ff9940a9a40 tx=0x7ff994006ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:19.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.557+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:07:19.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.557+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081af0 con 0x7ff980051540 2021-03-09T10:07:19.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.561+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99405d090 con 0x7ff980051540 2021-03-09T10:07:19.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.561+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:19.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.561+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:19.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:19.561+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:24.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800937d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:24.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800937d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:24.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800937d0 secure :-1 s=READY pgs=3657 cs=0 l=1 rev1=1 rx=0x7ff9940a9a40 tx=0x7ff994006ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:24.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800937d0 secure :-1 s=READY pgs=3657 cs=0 l=1 rev1=1 rx=0x7ff9940a9a40 tx=0x7ff994006ca0).stop 2021-03-09T10:07:24.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800937d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:24.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800937d0 unknown :-1 s=CLOSED pgs=3657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:24.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:24.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:24.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:24.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006020 con 0x7ff99c0adc20 2021-03-09T10:07:24.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800557f0 con 0x7ff99c0613a0 2021-03-09T10:07:24.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009d90 con 0x7ff980083100 2021-03-09T10:07:24.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:24.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:24.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:24.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:24.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.553+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:24.565 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:24.553+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12086 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:24.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.557+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:24.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.557+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:24.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.557+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:24.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.557+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:24.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.557+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054fe0 con 0x7ff980083100 2021-03-09T10:07:24.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.557+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003e310 con 0x7ff980083100 2021-03-09T10:07:24.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.557+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 rx=0x7ff98003d6a0 tx=0x7ff98c0a30d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:24.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.557+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075020 con 0x7ff980083100 2021-03-09T10:07:24.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.561+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c004070 con 0x7ff980083100 2021-03-09T10:07:24.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980083100 2021-03-09T10:07:24.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.565+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:24.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.565+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:24.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:24.565+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:29.134 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:07:29.138 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:07:29.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:29.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:29.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 rx=0x7ff98003d6a0 tx=0x7ff98c0a30d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:29.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 rx=0x7ff98003d6a0 tx=0x7ff98c0a30d0).stop 2021-03-09T10:07:29.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:29.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=CLOSED pgs=3669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:29.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:29.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:29.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:29.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800557f0 con 0x7ff99c0adc20 2021-03-09T10:07:29.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009d90 con 0x7ff99c0613a0 2021-03-09T10:07:29.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006020 con 0x7ff980051540 2021-03-09T10:07:29.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:29.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:29.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:29.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:29.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.557+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:29.569 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:29.557+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12112 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:29.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.561+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:29.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.561+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:29.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.561+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:29.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.561+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:29.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.561+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003bb50 con 0x7ff980051540 2021-03-09T10:07:29.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.561+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980028290 con 0x7ff980051540 2021-03-09T10:07:29.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.561+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 rx=0x7ff980024570 tx=0x7ff9940a9650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:29.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.561+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:07:29.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940a9880 con 0x7ff980051540 2021-03-09T10:07:29.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.569+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994081af0 con 0x7ff980051540 2021-03-09T10:07:29.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.569+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:29.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.569+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:29.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:29.569+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:34.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980043280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:34.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980043280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:34.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 rx=0x7ff980024570 tx=0x7ff9940a9650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:34.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 rx=0x7ff980024570 tx=0x7ff9940a9650).stop 2021-03-09T10:07:34.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980043280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:34.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 unknown :-1 s=CLOSED pgs=3685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:34.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:34.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:34.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:34.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056880 con 0x7ff99c0adc20 2021-03-09T10:07:34.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033b70 con 0x7ff99c0613a0 2021-03-09T10:07:34.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033cb0 con 0x7ff980083100 2021-03-09T10:07:34.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:34.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:34.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:34.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:34.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:34.571 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:34.561+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12147 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:34.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:34.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:34.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:34.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:34.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980033df0 con 0x7ff980083100 2021-03-09T10:07:34.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.561+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005870 con 0x7ff980083100 2021-03-09T10:07:34.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.565+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984003540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:34.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98402f000 con 0x7ff980083100 2021-03-09T10:07:34.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.569+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98402f000 con 0x7ff980083100 2021-03-09T10:07:34.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.573+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984025a40 con 0x7ff980083100 2021-03-09T10:07:34.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.573+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:34.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.573+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:34.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:34.573+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:39.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:39.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:39.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984003540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:39.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984003540).stop 2021-03-09T10:07:39.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:39.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=CLOSED pgs=3693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:39.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:39.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:39.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:39.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e580 con 0x7ff99c0adc20 2021-03-09T10:07:39.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004acd0 con 0x7ff99c0613a0 2021-03-09T10:07:39.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980028290 con 0x7ff980051540 2021-03-09T10:07:39.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:39.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:39.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:39.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:39.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:39.573 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:39.565+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12190 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:39.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:39.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:39.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:39.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:39.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800275a0 con 0x7ff980051540 2021-03-09T10:07:39.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980033df0 con 0x7ff980051540 2021-03-09T10:07:39.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 secure :-1 s=READY pgs=3700 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff98407c470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:39.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984082e50 con 0x7ff980051540 2021-03-09T10:07:39.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400b2a0 con 0x7ff980051540 2021-03-09T10:07:39.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.569+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b410 con 0x7ff980051540 2021-03-09T10:07:39.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.569+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:39.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.569+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:39.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:39.569+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:44.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980043280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:07:44.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980043280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:44.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 secure :-1 s=READY pgs=3700 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff98407c470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:44.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 secure :-1 s=READY pgs=3700 cs=0 l=1 rev1=1 rx=0x7ff984058aa0 tx=0x7ff98407c470).stop 2021-03-09T10:07:44.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980043280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:44.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043280 unknown :-1 s=CLOSED pgs=3700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:44.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:44.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:44.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:44.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:44.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:44.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056880 con 0x7ff99c0adc20 2021-03-09T10:07:44.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e5d0 con 0x7ff99c0613a0 2021-03-09T10:07:44.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f2c0 con 0x7ff980083100 2021-03-09T10:07:44.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:44.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:44.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:44.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:44.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:44.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:44.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:44.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003f430 con 0x7ff980083100 2021-03-09T10:07:44.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003bb50 con 0x7ff980083100 2021-03-09T10:07:44.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.565+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 rx=0x7ff980033800 tx=0x7ff9940a9730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:44.583 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:44.565+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12219 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:44.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.569+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980083100 2021-03-09T10:07:44.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.569+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99406fda0 con 0x7ff980083100 2021-03-09T10:07:44.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.569+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:07:44.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.569+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:44.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.569+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:44.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:44.569+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:49.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:49.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:49.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 rx=0x7ff980033800 tx=0x7ff9940a9730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:49.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 rx=0x7ff980033800 tx=0x7ff9940a9730).stop 2021-03-09T10:07:49.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:49.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=CLOSED pgs=3711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:49.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:49.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:49.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:49.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011270 con 0x7ff99c0adc20 2021-03-09T10:07:49.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056a20 con 0x7ff99c0613a0 2021-03-09T10:07:49.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003bb50 con 0x7ff980051540 2021-03-09T10:07:49.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:49.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:49.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:49.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:49.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.569+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:49.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.573+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:49.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.573+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:49.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.573+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:49.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.573+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:49.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.573+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003f2a0 con 0x7ff980051540 2021-03-09T10:07:49.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.573+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f430 con 0x7ff980051540 2021-03-09T10:07:49.581 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:49.573+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12258 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:49.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.573+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 rx=0x7ff9940b0040 tx=0x7ff99405d170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:49.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994012c80 con 0x7ff980051540 2021-03-09T10:07:49.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff980051540 2021-03-09T10:07:49.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994040000 con 0x7ff980051540 2021-03-09T10:07:49.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.577+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:49.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.581+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:49.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:49.581+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:54.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800935f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:54.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800935f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:54.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 rx=0x7ff9940b0040 tx=0x7ff99405d170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:54.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 rx=0x7ff9940b0040 tx=0x7ff99405d170).stop 2021-03-09T10:07:54.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800935f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:54.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800935f0 unknown :-1 s=CLOSED pgs=3721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:54.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:54.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:54.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:54.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055da0 con 0x7ff99c0adc20 2021-03-09T10:07:54.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088c50 con 0x7ff99c0613a0 2021-03-09T10:07:54.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009d90 con 0x7ff980083100 2021-03-09T10:07:54.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:54.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:54.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:54.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:54.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.573+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:54.582 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:54.573+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12302 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:54.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.577+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:54.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.577+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:54.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.577+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:54.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.577+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:54.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.577+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980028290 con 0x7ff980083100 2021-03-09T10:07:54.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.577+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054fe0 con 0x7ff980083100 2021-03-09T10:07:54.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.577+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 rx=0x7ff98c07d2b0 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:54.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075020 con 0x7ff980083100 2021-03-09T10:07:54.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980083100 2021-03-09T10:07:54.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980083100 2021-03-09T10:07:54.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.585+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:54.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.585+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:54.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:54.585+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:07:59.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:07:59.209 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:07:59.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:59.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:59.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 rx=0x7ff98c07d2b0 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:59.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 rx=0x7ff98c07d2b0 tx=0x7ff98c003e80).stop 2021-03-09T10:07:59.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800138b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:59.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800138b0 unknown :-1 s=CLOSED pgs=3732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:59.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800339d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:59.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:59.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:59.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009d90 con 0x7ff99c0adc20 2021-03-09T10:07:59.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056a20 con 0x7ff99c0613a0 2021-03-09T10:07:59.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054fe0 con 0x7ff980051540 2021-03-09T10:07:59.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:07:59.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:07:59.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:59.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:59.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.577+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:59.585 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:07:59.577+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12330 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:07:59.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.581+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:59.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.581+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:59.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.581+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:59.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.581+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:59.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.581+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088c50 con 0x7ff980051540 2021-03-09T10:07:59.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.581+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800275a0 con 0x7ff980051540 2021-03-09T10:07:59.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.581+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800339d0 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 rx=0x7ff98409b320 tx=0x7ff984046340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:59.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.581+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98402a000 con 0x7ff980051540 2021-03-09T10:07:59.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.589+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984025c10 con 0x7ff980051540 2021-03-09T10:07:59.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.589+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b410 con 0x7ff980051540 2021-03-09T10:07:59.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.589+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:59.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.589+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:59.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:07:59.589+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:04.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:04.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:04.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800339d0 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 rx=0x7ff98409b320 tx=0x7ff984046340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:04.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800339d0 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 rx=0x7ff98409b320 tx=0x7ff984046340).stop 2021-03-09T10:08:04.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800339d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:04.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800339d0 unknown :-1 s=CLOSED pgs=3749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:04.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:04.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:04.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:04.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038590 con 0x7ff99c0adc20 2021-03-09T10:08:04.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013530 con 0x7ff99c0613a0 2021-03-09T10:08:04.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009d90 con 0x7ff980083100 2021-03-09T10:08:04.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:04.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:04.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:04.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.577+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:04.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.581+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:04.587 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:04.581+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12363 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:04.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.581+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:04.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.581+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:04.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.581+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:04.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.581+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:04.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.585+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054fe0 con 0x7ff980083100 2021-03-09T10:08:04.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.585+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980028290 con 0x7ff980083100 2021-03-09T10:08:04.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.585+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003aac0 secure :-1 s=READY pgs=3758 cs=0 l=1 rev1=1 rx=0x7ff98c020480 tx=0x7ff98c046760).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:04.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980083100 2021-03-09T10:08:04.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.589+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:08:04.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.589+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:08:04.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.589+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:04.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.593+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:04.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:04.593+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:09.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:09.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:09.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003aac0 secure :-1 s=READY pgs=3758 cs=0 l=1 rev1=1 rx=0x7ff98c020480 tx=0x7ff98c046760).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:09.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003aac0 secure :-1 s=READY pgs=3758 cs=0 l=1 rev1=1 rx=0x7ff98c020480 tx=0x7ff98c046760).stop 2021-03-09T10:08:09.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003aac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:09.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003aac0 unknown :-1 s=CLOSED pgs=3758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:09.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:09.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:09.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:09.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:09.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009600 con 0x7ff99c0adc20 2021-03-09T10:08:09.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:09.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039330 con 0x7ff99c0613a0 2021-03-09T10:08:09.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980028290 con 0x7ff980051540 2021-03-09T10:08:09.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:09.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:09.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:09.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:09.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:09.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:09.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:09.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800275a0 con 0x7ff980051540 2021-03-09T10:08:09.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.581+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088c50 con 0x7ff980051540 2021-03-09T10:08:09.592 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:09.581+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12402 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:09.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.585+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093460 secure :-1 s=READY pgs=3764 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff984035560).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:09.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984018b90 con 0x7ff980051540 2021-03-09T10:08:09.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400b380 con 0x7ff980051540 2021-03-09T10:08:09.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.589+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98400b380 con 0x7ff980051540 2021-03-09T10:08:09.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.589+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:09.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.589+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:09.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:09.589+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:14.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:14.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:14.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093460 secure :-1 s=READY pgs=3764 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff984035560).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:14.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093460 secure :-1 s=READY pgs=3764 cs=0 l=1 rev1=1 rx=0x7ff984000e60 tx=0x7ff984035560).stop 2021-03-09T10:08:14.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:14.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093460 unknown :-1 s=CLOSED pgs=3764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:14.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:14.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:14.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:14.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009600 con 0x7ff99c0adc20 2021-03-09T10:08:14.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980034060 con 0x7ff99c0613a0 2021-03-09T10:08:14.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d360 con 0x7ff980083100 2021-03-09T10:08:14.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:14.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:14.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:14.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:14.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.585+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:14.593 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:14.585+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12444 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:14.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.589+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:14.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.589+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:14.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.589+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:14.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.589+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:14.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.589+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013c10 con 0x7ff980083100 2021-03-09T10:08:14.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.589+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800275a0 con 0x7ff980083100 2021-03-09T10:08:14.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.589+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093950 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 rx=0x7ff98400f220 tx=0x7ff9840238f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:14.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.589+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984046000 con 0x7ff980083100 2021-03-09T10:08:14.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984056e20 con 0x7ff980083100 2021-03-09T10:08:14.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984046000 con 0x7ff980083100 2021-03-09T10:08:14.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.597+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:14.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.597+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:14.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:14.597+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:19.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:19.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:19.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093950 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 rx=0x7ff98400f220 tx=0x7ff9840238f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:19.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093950 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 rx=0x7ff98400f220 tx=0x7ff9840238f0).stop 2021-03-09T10:08:19.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:19.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093950 unknown :-1 s=CLOSED pgs=3778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:19.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:19.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:19.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:19.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009600 con 0x7ff99c0adc20 2021-03-09T10:08:19.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056a20 con 0x7ff99c0613a0 2021-03-09T10:08:19.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.585+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800251d0 con 0x7ff980051540 2021-03-09T10:08:19.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:19.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:19.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:19.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:19.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:19.598 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:19.589+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12482 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:19.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:19.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:19.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:19.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:19.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980028290 con 0x7ff980051540 2021-03-09T10:08:19.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054fe0 con 0x7ff980051540 2021-03-09T10:08:19.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.589+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a5c0 secure :-1 s=READY pgs=3785 cs=0 l=1 rev1=1 rx=0x7ff98c099cb0 tx=0x7ff98c0040e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:19.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c099af0 con 0x7ff980051540 2021-03-09T10:08:19.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0160c0 con 0x7ff980051540 2021-03-09T10:08:19.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:08:19.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.597+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:19.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.597+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:19.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:19.597+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:24.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:24.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:24.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a5c0 secure :-1 s=READY pgs=3785 cs=0 l=1 rev1=1 rx=0x7ff98c099cb0 tx=0x7ff98c0040e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:24.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a5c0 secure :-1 s=READY pgs=3785 cs=0 l=1 rev1=1 rx=0x7ff98c099cb0 tx=0x7ff98c0040e0).stop 2021-03-09T10:08:24.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:24.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a5c0 unknown :-1 s=CLOSED pgs=3785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:24.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:24.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:24.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:24.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e770 con 0x7ff99c0adc20 2021-03-09T10:08:24.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054fe0 con 0x7ff99c0613a0 2021-03-09T10:08:24.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056b60 con 0x7ff980083100 2021-03-09T10:08:24.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:24.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:24.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:24.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:24.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.589+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:24.598 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:24.589+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12515 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:24.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.593+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:24.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.593+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:24.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.593+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:24.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.593+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:24.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.593+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980056ca0 con 0x7ff980083100 2021-03-09T10:08:24.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.593+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013c10 con 0x7ff980083100 2021-03-09T10:08:24.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.593+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088ab0 secure :-1 s=READY pgs=3799 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984003530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:24.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98407dcd0 con 0x7ff980083100 2021-03-09T10:08:24.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98407dcd0 con 0x7ff980083100 2021-03-09T10:08:24.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98402fc20 con 0x7ff980083100 2021-03-09T10:08:24.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.601+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:24.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.601+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:24.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:24.601+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:29.253 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:08:29.256 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:08:29.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980088ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:29.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980088ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:29.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088ab0 secure :-1 s=READY pgs=3799 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984003530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:29.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088ab0 secure :-1 s=READY pgs=3799 cs=0 l=1 rev1=1 rx=0x7ff98407b700 tx=0x7ff984003530).stop 2021-03-09T10:08:29.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980088ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:29.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980088ab0 unknown :-1 s=CLOSED pgs=3799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:29.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:29.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:29.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:29.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800937e0 con 0x7ff99c0adc20 2021-03-09T10:08:29.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002f9e0 con 0x7ff99c0613a0 2021-03-09T10:08:29.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051ac0 con 0x7ff980051540 2021-03-09T10:08:29.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:29.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:29.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:29.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:29.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:29.600 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:29.593+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12550 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:29.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:29.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:29.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:29.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:29.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054fe0 con 0x7ff980051540 2021-03-09T10:08:29.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980028290 con 0x7ff980051540 2021-03-09T10:08:29.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093950 secure :-1 s=READY pgs=3807 cs=0 l=1 rev1=1 rx=0x7ff980044a50 tx=0x7ff98c08e180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:29.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980051540 2021-03-09T10:08:29.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T10:08:29.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980051540 2021-03-09T10:08:29.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.601+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:29.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.601+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:29.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:29.601+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:34.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:34.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:34.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093950 secure :-1 s=READY pgs=3807 cs=0 l=1 rev1=1 rx=0x7ff980044a50 tx=0x7ff98c08e180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:34.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093950 secure :-1 s=READY pgs=3807 cs=0 l=1 rev1=1 rx=0x7ff980044a50 tx=0x7ff98c08e180).stop 2021-03-09T10:08:34.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:34.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093950 unknown :-1 s=CLOSED pgs=3807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:34.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:34.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:34.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:34.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003ce70 con 0x7ff99c0adc20 2021-03-09T10:08:34.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800553e0 con 0x7ff99c0613a0 2021-03-09T10:08:34.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800066f0 con 0x7ff980083100 2021-03-09T10:08:34.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:34.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:34.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:34.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:34.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:34.603 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:34.597+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12575 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:34.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.597+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:34.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.597+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:34.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.597+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:34.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.597+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:34.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.597+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980028290 con 0x7ff980083100 2021-03-09T10:08:34.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.597+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980056ca0 con 0x7ff980083100 2021-03-09T10:08:34.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.601+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003a0c0 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 rx=0x7ff98400f3d0 tx=0x7ff984000d00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:34.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.605+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98402fdf0 con 0x7ff980083100 2021-03-09T10:08:34.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.605+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840812f0 con 0x7ff980083100 2021-03-09T10:08:34.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.605+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff984025a40 con 0x7ff980083100 2021-03-09T10:08:34.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.605+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:34.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.605+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:34.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:34.605+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:39.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:39.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:39.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003a0c0 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 rx=0x7ff98400f3d0 tx=0x7ff984000d00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:39.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003a0c0 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 rx=0x7ff98400f3d0 tx=0x7ff984000d00).stop 2021-03-09T10:08:39.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003a0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:39.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003a0c0 unknown :-1 s=CLOSED pgs=3821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:39.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:39.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:39.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:39.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800553e0 con 0x7ff99c0adc20 2021-03-09T10:08:39.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a8e0 con 0x7ff99c0613a0 2021-03-09T10:08:39.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037940 con 0x7ff980051540 2021-03-09T10:08:39.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:39.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:39.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:39.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:39.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.597+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:39.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:39.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:39.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:39.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:39.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a750 con 0x7ff980051540 2021-03-09T10:08:39.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f2a0 con 0x7ff980051540 2021-03-09T10:08:39.608 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:39.601+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12625 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:39.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088ab0 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 rx=0x7ff980013720 tx=0x7ff994081670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:39.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980051540 2021-03-09T10:08:39.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400bed0 con 0x7ff980051540 2021-03-09T10:08:39.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.605+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff980051540 2021-03-09T10:08:39.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.605+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:39.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.605+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:39.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:39.605+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:44.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:44.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:44.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088ab0 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 rx=0x7ff980013720 tx=0x7ff994081670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:44.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088ab0 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 rx=0x7ff980013720 tx=0x7ff994081670).stop 2021-03-09T10:08:44.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:44.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088ab0 unknown :-1 s=CLOSED pgs=3828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:44.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:44.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:44.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:44.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:44.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:44.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039ed0 con 0x7ff99c0adc20 2021-03-09T10:08:44.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a010 con 0x7ff99c0613a0 2021-03-09T10:08:44.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a150 con 0x7ff980083100 2021-03-09T10:08:44.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:44.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:44.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:44.609 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:44.601+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12662 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:44.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.605+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:44.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.605+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:44.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.605+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:44.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.605+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:44.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.605+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a2c0 con 0x7ff980083100 2021-03-09T10:08:44.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.605+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003a750 con 0x7ff980083100 2021-03-09T10:08:44.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.605+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003f0b0 secure :-1 s=READY pgs=3843 cs=0 l=1 rev1=1 rx=0x7ff994062010 tx=0x7ff99408b6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:44.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.605+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980083100 2021-03-09T10:08:44.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.613+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a4b0 con 0x7ff980083100 2021-03-09T10:08:44.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.613+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff980083100 2021-03-09T10:08:44.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.613+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:44.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.613+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:44.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:44.613+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:49.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.600+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:49.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.600+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:49.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.600+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003f0b0 secure :-1 s=READY pgs=3843 cs=0 l=1 rev1=1 rx=0x7ff994062010 tx=0x7ff99408b6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:49.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.600+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003f0b0 secure :-1 s=READY pgs=3843 cs=0 l=1 rev1=1 rx=0x7ff994062010 tx=0x7ff99408b6c0).stop 2021-03-09T10:08:49.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.600+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98003f0b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:49.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.600+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98003f0b0 unknown :-1 s=CLOSED pgs=3843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:49.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:49.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:49.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:49.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037f70 con 0x7ff99c0adc20 2021-03-09T10:08:49.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050350 con 0x7ff99c0613a0 2021-03-09T10:08:49.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800275a0 con 0x7ff980051540 2021-03-09T10:08:49.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:49.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:49.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:49.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:49.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:49.612 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:49.604+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12689 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:49.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:49.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:49.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:49.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:49.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093950 con 0x7ff980051540 2021-03-09T10:08:49.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003a2c0 con 0x7ff980051540 2021-03-09T10:08:49.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a60 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 rx=0x7ff98000aa20 tx=0x7ff99407b980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:49.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.604+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408b880 con 0x7ff980051540 2021-03-09T10:08:49.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.608+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408b880 con 0x7ff980051540 2021-03-09T10:08:49.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.612+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940130c0 con 0x7ff980051540 2021-03-09T10:08:49.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.612+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:49.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.612+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:49.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.612+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff98003bce0 msgr2=0x7ff980021c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:08:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff98003bce0 msgr2=0x7ff980021c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:49.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff98003bce0 0x7ff980021c10 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7ff99407c040 tx=0x7ff994017000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:49.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff98003bce0 0x7ff980021c10 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7ff99407c040 tx=0x7ff994017000).stop 2021-03-09T10:08:49.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff98003bce0 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:08:49.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff9937fe700 4 mgrc ms_handle_reset ms_handle_reset con 0x7ff98003bce0 2021-03-09T10:08:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff9937fe700 4 mgrc reconnect Terminating session with v2:172.21.15.52:6808/14433 2021-03-09T10:08:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff98003bce0 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff98003bce0 0x7ff980021c10 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff9937fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:49.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff980039e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:49.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff980039e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:49.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:49.892+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff980039e80 secure :-1 s=READY pgs=3675 cs=0 l=1 rev1=1 rx=0x7ff984003940 tx=0x7ff984000d00).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:54.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:54.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:54.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a60 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 rx=0x7ff98000aa20 tx=0x7ff99407b980).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:54.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a60 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 rx=0x7ff98000aa20 tx=0x7ff99407b980).stop 2021-03-09T10:08:54.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:54.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a60 unknown :-1 s=CLOSED pgs=3849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:54.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:54.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:54.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:54.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800275a0 con 0x7ff99c0adc20 2021-03-09T10:08:54.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d440 con 0x7ff99c0613a0 2021-03-09T10:08:54.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a370 con 0x7ff98003bce0 2021-03-09T10:08:54.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:54.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:54.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:54.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:54.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.604+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:54.614 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:54.608+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12734 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:54.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.608+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:54.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.608+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:54.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.608+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:54.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.608+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:54.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.608+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093ae0 con 0x7ff98003bce0 2021-03-09T10:08:54.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.608+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054fe0 con 0x7ff98003bce0 2021-03-09T10:08:54.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.612+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010a40 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 rx=0x7ff98c08e240 tx=0x7ff98c003f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:54.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.612+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075020 con 0x7ff98003bce0 2021-03-09T10:08:54.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.612+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08f300 con 0x7ff98003bce0 2021-03-09T10:08:54.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.616+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98003bce0 2021-03-09T10:08:54.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.616+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:54.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.616+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:54.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:54.616+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:08:59.316 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:08:59.319 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:08:59.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980010a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:59.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980010a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:59.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010a40 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 rx=0x7ff98c08e240 tx=0x7ff98c003f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:59.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010a40 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 rx=0x7ff98c08e240 tx=0x7ff98c003f00).stop 2021-03-09T10:08:59.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980010a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:59.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980010a40 unknown :-1 s=CLOSED pgs=3864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:59.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:59.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:59.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:59.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:08:59.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800433a0 con 0x7ff99c0613a0 2021-03-09T10:08:59.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027570 con 0x7ff980051540 2021-03-09T10:08:59.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:08:59.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:08:59.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:59.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:59.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.608+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:59.617 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:08:59.608+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12763 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:08:59.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:59.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:59.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:59.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:59.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980037d80 con 0x7ff980051540 2021-03-09T10:08:59.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093950 con 0x7ff980051540 2021-03-09T10:08:59.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a10 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff99408b6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:59.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.612+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994065d60 con 0x7ff980051540 2021-03-09T10:08:59.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.612+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994066880 con 0x7ff980051540 2021-03-09T10:08:59.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:08:59.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.620+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:59.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.620+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:59.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:08:59.620+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:04.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:04.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:04.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a10 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff99408b6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:04.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a10 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff99408b6c0).stop 2021-03-09T10:09:04.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088a10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:04.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088a10 unknown :-1 s=CLOSED pgs=3874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:04.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:04.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:04.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980093c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:04.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:09:04.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800433a0 con 0x7ff99c0613a0 2021-03-09T10:09:04.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093950 con 0x7ff98003bce0 2021-03-09T10:09:04.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:04.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:04.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:04.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.608+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:04.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980093c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:04.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:04.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:04.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:04.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:04.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054fe0 con 0x7ff98003bce0 2021-03-09T10:09:04.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980037d80 con 0x7ff98003bce0 2021-03-09T10:09:04.621 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:04.612+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12806 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:04.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980093c70 secure :-1 s=READY pgs=3881 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994066880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:04.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.616+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff98003bce0 2021-03-09T10:09:04.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.616+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff98003bce0 2021-03-09T10:09:04.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff98003bce0 2021-03-09T10:09:04.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.620+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:04.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.620+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:04.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:04.620+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:09.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980093c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:09.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980093c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:09.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980093c70 secure :-1 s=READY pgs=3881 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994066880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:09.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980093c70 secure :-1 s=READY pgs=3881 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994066880).stop 2021-03-09T10:09:09.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 msgr2=0x7ff980093c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:09.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98003bce0 0x7ff980093c70 unknown :-1 s=CLOSED pgs=3881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:09.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:09.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800889c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:09.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:09.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:09:09.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ffa0 con 0x7ff99c0613a0 2021-03-09T10:09:09.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001af50 con 0x7ff980051540 2021-03-09T10:09:09.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:09.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800889c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:09.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:09.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:09.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.612+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:09.622 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:09.616+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12838 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:09.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.616+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:09.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.616+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:09.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.616+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:09.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.616+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:09.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.616+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093920 con 0x7ff980051540 2021-03-09T10:09:09.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.616+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093ae0 con 0x7ff980051540 2021-03-09T10:09:09.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.616+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800889c0 secure :-1 s=READY pgs=3892 cs=0 l=1 rev1=1 rx=0x7ff98c046470 tx=0x7ff98c0464a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:09.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.616+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T10:09:09.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c099950 con 0x7ff980051540 2021-03-09T10:09:09.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.624+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:09:09.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.624+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:09.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.624+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:09.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:09.624+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:14.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800889c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:14.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800889c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:14.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800889c0 secure :-1 s=READY pgs=3892 cs=0 l=1 rev1=1 rx=0x7ff98c046470 tx=0x7ff98c0464a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:14.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800889c0 secure :-1 s=READY pgs=3892 cs=0 l=1 rev1=1 rx=0x7ff98c046470 tx=0x7ff98c0464a0).stop 2021-03-09T10:09:14.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800889c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:14.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800889c0 unknown :-1 s=CLOSED pgs=3892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:14.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:14.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:14.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:14.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:09:14.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ffa0 con 0x7ff99c0613a0 2021-03-09T10:09:14.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800433a0 con 0x7ff980050140 2021-03-09T10:09:14.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:14.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:14.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:14.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:14.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.616+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:14.626 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:14.616+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12870 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:14.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:14.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:14.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:14.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:14.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980050520 con 0x7ff980050140 2021-03-09T10:09:14.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054fe0 con 0x7ff980050140 2021-03-09T10:09:14.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 secure :-1 s=READY pgs=3909 cs=0 l=1 rev1=1 rx=0x7ff99402a000 tx=0x7ff99403e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:14.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994013eb0 con 0x7ff980050140 2021-03-09T10:09:14.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.624+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994013eb0 con 0x7ff980050140 2021-03-09T10:09:14.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:09:14.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.628+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:14.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.628+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:14.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:14.628+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:19.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:19.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:19.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 secure :-1 s=READY pgs=3909 cs=0 l=1 rev1=1 rx=0x7ff99402a000 tx=0x7ff99403e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:19.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 secure :-1 s=READY pgs=3909 cs=0 l=1 rev1=1 rx=0x7ff99402a000 tx=0x7ff99403e000).stop 2021-03-09T10:09:19.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093b50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:19.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 unknown :-1 s=CLOSED pgs=3909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:19.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:19.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:19.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:19.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:09:19.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001af50 con 0x7ff99c0613a0 2021-03-09T10:09:19.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008dd20 con 0x7ff980051540 2021-03-09T10:09:19.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:19.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:19.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:19.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:19.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:19.628 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:19.620+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12912 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:19.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:19.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:19.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:19.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:19.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054fe0 con 0x7ff980051540 2021-03-09T10:09:19.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980050520 con 0x7ff980051540 2021-03-09T10:09:19.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.624+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088920 secure :-1 s=READY pgs=3915 cs=0 l=1 rev1=1 rx=0x7ff99405d400 tx=0x7ff994029200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:19.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069030 con 0x7ff980051540 2021-03-09T10:09:19.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994013870 con 0x7ff980051540 2021-03-09T10:09:19.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:09:19.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.628+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:19.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.632+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:19.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:19.632+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:24.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:24.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:24.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088920 secure :-1 s=READY pgs=3915 cs=0 l=1 rev1=1 rx=0x7ff99405d400 tx=0x7ff994029200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:24.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088920 secure :-1 s=READY pgs=3915 cs=0 l=1 rev1=1 rx=0x7ff99405d400 tx=0x7ff994029200).stop 2021-03-09T10:09:24.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980088920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:24.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980088920 unknown :-1 s=CLOSED pgs=3915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:24.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:24.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:24.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:24.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:09:24.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ffa0 con 0x7ff99c0613a0 2021-03-09T10:09:24.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000eb30 con 0x7ff980050140 2021-03-09T10:09:24.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:24.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:24.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:24.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:24.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.620+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:24.630 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:24.620+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12943 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:24.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:24.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:24.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:24.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:24.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003d280 con 0x7ff980050140 2021-03-09T10:09:24.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093920 con 0x7ff980050140 2021-03-09T10:09:24.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 secure :-1 s=READY pgs=3924 cs=0 l=1 rev1=1 rx=0x7ff98c074f90 tx=0x7ff98c00bef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:24.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c07de90 con 0x7ff980050140 2021-03-09T10:09:24.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07dcb0 con 0x7ff980050140 2021-03-09T10:09:24.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0469b0 con 0x7ff980050140 2021-03-09T10:09:24.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.628+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:24.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.628+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:24.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:24.628+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:29.580 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:09:29.583 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:09:29.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:29.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:29.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 secure :-1 s=READY pgs=3924 cs=0 l=1 rev1=1 rx=0x7ff98c074f90 tx=0x7ff98c00bef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:29.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 secure :-1 s=READY pgs=3924 cs=0 l=1 rev1=1 rx=0x7ff98c074f90 tx=0x7ff98c00bef0).stop 2021-03-09T10:09:29.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093b50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:29.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093b50 unknown :-1 s=CLOSED pgs=3924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:29.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:29.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:29.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:29.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:09:29.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004aec0 con 0x7ff99c0613a0 2021-03-09T10:09:29.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008dec0 con 0x7ff980051540 2021-03-09T10:09:29.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:29.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:29.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:29.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:29.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:29.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:29.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:29.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:29.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:29.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000eb30 con 0x7ff980051540 2021-03-09T10:09:29.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003d280 con 0x7ff980051540 2021-03-09T10:09:29.633 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:29.624+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12973 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:29.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094040 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 rx=0x7ff98c0522a0 tx=0x7ff98c0a0f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:29.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c046b90 con 0x7ff980051540 2021-03-09T10:09:29.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c099930 con 0x7ff980051540 2021-03-09T10:09:29.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072970 con 0x7ff980051540 2021-03-09T10:09:29.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.628+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:29.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.632+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:29.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:29.632+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:34.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:34.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:34.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094040 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 rx=0x7ff98c0522a0 tx=0x7ff98c0a0f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:34.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094040 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 rx=0x7ff98c0522a0 tx=0x7ff98c0a0f50).stop 2021-03-09T10:09:34.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:34.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094040 unknown :-1 s=CLOSED pgs=3935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:34.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:34.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:34.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:34.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:09:34.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004aec0 con 0x7ff99c0613a0 2021-03-09T10:09:34.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f740 con 0x7ff980050140 2021-03-09T10:09:34.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:34.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:34.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:34.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:34.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:34.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:34.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:34.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:34.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:34.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980003a40 con 0x7ff980050140 2021-03-09T10:09:34.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.624+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054fe0 con 0x7ff980050140 2021-03-09T10:09:34.636 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:34.628+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13010 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:34.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.628+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 secure :-1 s=READY pgs=3945 cs=0 l=1 rev1=1 rx=0x7ff994068810 tx=0x7ff994029190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:34.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407aa60 con 0x7ff980050140 2021-03-09T10:09:34.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940a2350 con 0x7ff980050140 2021-03-09T10:09:34.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.632+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:09:34.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.632+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:34.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.632+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:34.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:34.632+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:39.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980027320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:39.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980027320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:39.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 secure :-1 s=READY pgs=3945 cs=0 l=1 rev1=1 rx=0x7ff994068810 tx=0x7ff994029190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:39.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 secure :-1 s=READY pgs=3945 cs=0 l=1 rev1=1 rx=0x7ff994068810 tx=0x7ff994029190).stop 2021-03-09T10:09:39.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980027320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:39.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 unknown :-1 s=CLOSED pgs=3945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:39.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:39.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:39.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:39.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027e40 con 0x7ff99c0adc20 2021-03-09T10:09:39.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013500 con 0x7ff99c0613a0 2021-03-09T10:09:39.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f740 con 0x7ff980051540 2021-03-09T10:09:39.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:39.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:39.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:39.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:39.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.628+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:39.636 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:39.628+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13047 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:39.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:39.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:39.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:39.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:39.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093920 con 0x7ff980051540 2021-03-09T10:09:39.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980003a40 con 0x7ff980051540 2021-03-09T10:09:39.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013840 secure :-1 s=READY pgs=3960 cs=0 l=1 rev1=1 rx=0x7ff99407ae90 tx=0x7ff994064c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:39.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069030 con 0x7ff980051540 2021-03-09T10:09:39.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400c150 con 0x7ff980051540 2021-03-09T10:09:39.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.640+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:09:39.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.640+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:39.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.640+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:39.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:39.640+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:44.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:44.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:44.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013840 secure :-1 s=READY pgs=3960 cs=0 l=1 rev1=1 rx=0x7ff99407ae90 tx=0x7ff994064c50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:44.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013840 secure :-1 s=READY pgs=3960 cs=0 l=1 rev1=1 rx=0x7ff99407ae90 tx=0x7ff994064c50).stop 2021-03-09T10:09:44.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:44.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013840 unknown :-1 s=CLOSED pgs=3960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:44.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:44.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:44.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:44.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033250 con 0x7ff99c0adc20 2021-03-09T10:09:44.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030420 con 0x7ff99c0613a0 2021-03-09T10:09:44.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013d30 con 0x7ff980050140 2021-03-09T10:09:44.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:44.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:44.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:44.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:44.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:44.639 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:44.632+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13084 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:44.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:44.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:44.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:44.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:44.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003f740 con 0x7ff980050140 2021-03-09T10:09:44.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093920 con 0x7ff980050140 2021-03-09T10:09:44.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.636+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093ab0 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 rx=0x7ff99409e060 tx=0x7ff99405db50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:44.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980050140 2021-03-09T10:09:44.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407aa60 con 0x7ff980050140 2021-03-09T10:09:44.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.640+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:09:44.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.640+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:44.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.640+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:44.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:44.640+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:49.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:49.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:49.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093ab0 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 rx=0x7ff99409e060 tx=0x7ff99405db50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:49.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093ab0 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 rx=0x7ff99409e060 tx=0x7ff99405db50).stop 2021-03-09T10:09:49.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:49.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093ab0 unknown :-1 s=CLOSED pgs=3967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:49.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:49.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:49.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:49.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033280 con 0x7ff99c0adc20 2021-03-09T10:09:49.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003da00 con 0x7ff99c0613a0 2021-03-09T10:09:49.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800433c0 con 0x7ff980051540 2021-03-09T10:09:49.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:49.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:49.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:49.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:49.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:49.641 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:49.632+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13117 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:49.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:49.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:49.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:49.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:49.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054fe0 con 0x7ff980051540 2021-03-09T10:09:49.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.632+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f740 con 0x7ff980051540 2021-03-09T10:09:49.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.636+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fa0 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 rx=0x7ff980056be0 tx=0x7ff99403ef40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:49.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.640+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:09:49.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.640+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994064840 con 0x7ff980051540 2021-03-09T10:09:49.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.640+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:09:49.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.640+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:49.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.640+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:49.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:49.640+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:54.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:54.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:54.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fa0 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 rx=0x7ff980056be0 tx=0x7ff99403ef40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:54.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fa0 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 rx=0x7ff980056be0 tx=0x7ff99403ef40).stop 2021-03-09T10:09:54.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:54.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fa0 unknown :-1 s=CLOSED pgs=3977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:54.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:54.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:54.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:54.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015d20 con 0x7ff99c0adc20 2021-03-09T10:09:54.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0613a0 2021-03-09T10:09:54.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800314e0 con 0x7ff980050140 2021-03-09T10:09:54.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:54.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:54.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:54.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:54.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:54.644 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:54.636+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13156 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:54.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:54.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:54.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:54.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:54.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013520 con 0x7ff980050140 2021-03-09T10:09:54.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000eb30 con 0x7ff980050140 2021-03-09T10:09:54.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.636+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093920 secure :-1 s=READY pgs=3990 cs=0 l=1 rev1=1 rx=0x7ff980057030 tx=0x7ff98c006700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:54.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.644+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980050140 2021-03-09T10:09:54.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.644+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980050140 2021-03-09T10:09:54.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.644+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980050140 2021-03-09T10:09:54.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.644+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:54.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.644+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:54.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:54.644+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:09:59.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:09:59.622 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:09:59.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:59.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:59.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093920 secure :-1 s=READY pgs=3990 cs=0 l=1 rev1=1 rx=0x7ff980057030 tx=0x7ff98c006700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:59.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093920 secure :-1 s=READY pgs=3990 cs=0 l=1 rev1=1 rx=0x7ff980057030 tx=0x7ff98c006700).stop 2021-03-09T10:09:59.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:59.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093920 unknown :-1 s=CLOSED pgs=3990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:59.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:59.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:59.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:09:59.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:59.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:59.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:09:59.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051c60 con 0x7ff99c0613a0 2021-03-09T10:09:59.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003b1d0 con 0x7ff980051540 2021-03-09T10:09:59.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:09:59.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:59.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.636+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:59.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:59.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.640+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:59.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:59.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.640+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:59.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003f740 con 0x7ff980051540 2021-03-09T10:09:59.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054fe0 con 0x7ff980051540 2021-03-09T10:09:59.648 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:09:59.640+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13190 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:09:59.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.640+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fd0 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994066980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:59.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.640+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980051540 2021-03-09T10:09:59.650 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.644+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994092e10 con 0x7ff980051540 2021-03-09T10:09:59.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070020 con 0x7ff980051540 2021-03-09T10:09:59.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.648+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:59.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.648+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:59.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:09:59.648+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:04.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:04.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:04.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fd0 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994066980).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:04.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fd0 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994066980).stop 2021-03-09T10:10:04.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:04.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093fd0 unknown :-1 s=CLOSED pgs=4000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:04.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:04.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:04.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:04.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051e00 con 0x7ff99c0adc20 2021-03-09T10:10:04.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003850 con 0x7ff99c0613a0 2021-03-09T10:10:04.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:04.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003b1d0 con 0x7ff980050140 2021-03-09T10:10:04.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:04.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:04.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:04.650 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:04.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:04.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:04.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:04.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:04.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800456e0 con 0x7ff980050140 2021-03-09T10:10:04.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.640+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f740 con 0x7ff980050140 2021-03-09T10:10:04.652 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:04.644+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13240 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:04.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.644+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff99400c1c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:04.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980050140 2021-03-09T10:10:04.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940abe20 con 0x7ff980050140 2021-03-09T10:10:04.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff980050140 2021-03-09T10:10:04.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.648+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:04.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.648+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:04.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:04.648+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:09.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980027320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:09.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980027320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:09.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff99400c1c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:09.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff99400c1c0).stop 2021-03-09T10:10:09.650 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980027320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:09.650 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 unknown :-1 s=CLOSED pgs=4009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:09.650 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:09.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:09.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:09.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051e00 con 0x7ff99c0adc20 2021-03-09T10:10:09.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003850 con 0x7ff99c0613a0 2021-03-09T10:10:09.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003b1d0 con 0x7ff980051540 2021-03-09T10:10:09.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:09.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:09.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:09.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:09.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.644+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:09.654 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:09.644+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13263 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:09.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:09.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:09.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:09.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:09.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003f740 con 0x7ff980051540 2021-03-09T10:10:09.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800456e0 con 0x7ff980051540 2021-03-09T10:10:09.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994034230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:09.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff980051540 2021-03-09T10:10:09.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.656+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff980051540 2021-03-09T10:10:09.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.656+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:10:09.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.656+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:09.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.656+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:09.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:09.656+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:14.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:14.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:14.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994034230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:14.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994034230).stop 2021-03-09T10:10:14.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:14.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 unknown :-1 s=CLOSED pgs=4025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:14.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:14.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:14.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:14.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051e00 con 0x7ff99c0adc20 2021-03-09T10:10:14.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003b1d0 con 0x7ff99c0613a0 2021-03-09T10:10:14.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003850 con 0x7ff980050140 2021-03-09T10:10:14.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:14.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:14.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:14.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:14.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:14.655 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:14.648+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13308 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:14.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:14.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:14.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:14.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:14.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035fe0 con 0x7ff980050140 2021-03-09T10:10:14.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f740 con 0x7ff980050140 2021-03-09T10:10:14.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.652+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 rx=0x7ff99405e040 tx=0x7ff994081770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:14.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.652+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980050140 2021-03-09T10:10:14.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.652+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980050140 2021-03-09T10:10:14.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.656+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980050140 2021-03-09T10:10:14.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.656+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:14.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.660+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:14.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:14.660+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:19.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980027320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:10:19.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980027320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:19.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 rx=0x7ff99405e040 tx=0x7ff994081770).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:19.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 rx=0x7ff99405e040 tx=0x7ff994081770).stop 2021-03-09T10:10:19.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980027320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:19.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980027320 unknown :-1 s=CLOSED pgs=4033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:19.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:19.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:19.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:19.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051e00 con 0x7ff99c0adc20 2021-03-09T10:10:19.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003b1d0 con 0x7ff99c0613a0 2021-03-09T10:10:19.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800095c0 con 0x7ff980051540 2021-03-09T10:10:19.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:19.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:19.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:19.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:19.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.648+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:19.658 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:19.652+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13350 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:19.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.652+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:19.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.652+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:19.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.652+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:19.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.652+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:19.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.652+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035fe0 con 0x7ff980051540 2021-03-09T10:10:19.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.652+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013520 con 0x7ff980051540 2021-03-09T10:10:19.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.652+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 rx=0x7ff98c07d2b0 tx=0x7ff98c07d160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:19.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.656+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980051540 2021-03-09T10:10:19.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.656+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T10:10:19.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.660+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T10:10:19.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.660+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:19.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.660+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:19.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:19.660+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:24.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:24.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:24.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 rx=0x7ff98c07d2b0 tx=0x7ff98c07d160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:24.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 rx=0x7ff98c07d2b0 tx=0x7ff98c07d160).stop 2021-03-09T10:10:24.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980093920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:24.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980093920 unknown :-1 s=CLOSED pgs=4041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:24.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:24.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:24.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:24.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051e00 con 0x7ff99c0adc20 2021-03-09T10:10:24.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0613a0 2021-03-09T10:10:24.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800095c0 con 0x7ff980050140 2021-03-09T10:10:24.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:24.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:24.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:24.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:24.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:24.660 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:24.651+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13384 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:24.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:24.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:24.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:24.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:24.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035fe0 con 0x7ff980050140 2021-03-09T10:10:24.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.651+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f740 con 0x7ff980050140 2021-03-09T10:10:24.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.655+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093e10 secure :-1 s=READY pgs=4054 cs=0 l=1 rev1=1 rx=0x7ff9940320a0 tx=0x7ff99403ef10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:24.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020590 con 0x7ff980050140 2021-03-09T10:10:24.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.663+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980050140 2021-03-09T10:10:24.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.663+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:10:24.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.663+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:24.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.663+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:24.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:24.663+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:29.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:29.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:29.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093e10 secure :-1 s=READY pgs=4054 cs=0 l=1 rev1=1 rx=0x7ff9940320a0 tx=0x7ff99403ef10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:29.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093e10 secure :-1 s=READY pgs=4054 cs=0 l=1 rev1=1 rx=0x7ff9940320a0 tx=0x7ff99403ef10).stop 2021-03-09T10:10:29.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980093e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:29.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980093e10 unknown :-1 s=CLOSED pgs=4054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:29.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:29.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:29.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:29.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:29.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:10:29.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:29.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f8d0 con 0x7ff99c0613a0 2021-03-09T10:10:29.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800095c0 con 0x7ff980051540 2021-03-09T10:10:29.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:29.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:29.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:29.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:29.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:29.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:29.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:29.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035fe0 con 0x7ff980051540 2021-03-09T10:10:29.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013520 con 0x7ff980051540 2021-03-09T10:10:29.665 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:29.655+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13423 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:29.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.655+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094300 secure :-1 s=READY pgs=4065 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c0a9fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:29.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T10:10:29.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c052130 con 0x7ff980051540 2021-03-09T10:10:29.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.663+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c008d90 con 0x7ff980051540 2021-03-09T10:10:29.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.663+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:29.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.663+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:29.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:29.663+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:29.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:10:29.682 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:10:34.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:34.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:34.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094300 secure :-1 s=READY pgs=4065 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c0a9fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:34.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094300 secure :-1 s=READY pgs=4065 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c0a9fd0).stop 2021-03-09T10:10:34.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:34.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094300 unknown :-1 s=CLOSED pgs=4065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:34.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:34.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:34.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:34.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:10:34.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051e00 con 0x7ff99c0613a0 2021-03-09T10:10:34.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800095c0 con 0x7ff980050140 2021-03-09T10:10:34.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:34.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:34.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:34.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:34.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:34.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:34.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:34.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:34.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:34.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035fb0 con 0x7ff980050140 2021-03-09T10:10:34.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f740 con 0x7ff980050140 2021-03-09T10:10:34.671 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:34.659+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13444 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:34.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.659+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800947f0 secure :-1 s=READY pgs=4073 cs=0 l=1 rev1=1 rx=0x7ff994064e10 tx=0x7ff994078000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:34.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.663+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980050140 2021-03-09T10:10:34.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.663+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940aa420 con 0x7ff980050140 2021-03-09T10:10:34.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.667+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99409a020 con 0x7ff980050140 2021-03-09T10:10:34.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.667+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:34.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.667+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:34.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:34.667+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:39.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800947f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:39.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800947f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:39.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800947f0 secure :-1 s=READY pgs=4073 cs=0 l=1 rev1=1 rx=0x7ff994064e10 tx=0x7ff994078000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:39.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800947f0 secure :-1 s=READY pgs=4073 cs=0 l=1 rev1=1 rx=0x7ff994064e10 tx=0x7ff994078000).stop 2021-03-09T10:10:39.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800947f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:39.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800947f0 unknown :-1 s=CLOSED pgs=4073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:39.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:39.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:39.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:39.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051e00 con 0x7ff99c0adc20 2021-03-09T10:10:39.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046570 con 0x7ff99c0613a0 2021-03-09T10:10:39.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f2b0 con 0x7ff980051540 2021-03-09T10:10:39.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:39.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:39.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fc20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:39.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:39.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.659+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:39.670 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:39.663+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13495 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:39.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.663+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fc20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:39.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.663+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:39.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.663+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:39.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.663+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:39.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.663+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027320 con 0x7ff980051540 2021-03-09T10:10:39.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.663+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035fb0 con 0x7ff980051540 2021-03-09T10:10:39.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.663+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 secure :-1 s=READY pgs=4084 cs=0 l=1 rev1=1 rx=0x7ff98004b100 tx=0x7ff99405da60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:39.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.667+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:10:39.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.667+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980051540 2021-03-09T10:10:39.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.667+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:10:39.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.667+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:39.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.667+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:39.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:39.667+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:44.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:44.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:44.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 secure :-1 s=READY pgs=4084 cs=0 l=1 rev1=1 rx=0x7ff98004b100 tx=0x7ff99405da60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:44.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 secure :-1 s=READY pgs=4084 cs=0 l=1 rev1=1 rx=0x7ff98004b100 tx=0x7ff99405da60).stop 2021-03-09T10:10:44.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:44.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 unknown :-1 s=CLOSED pgs=4084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:44.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:44.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:44.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:44.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017450 con 0x7ff99c0adc20 2021-03-09T10:10:44.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f2b0 con 0x7ff99c0613a0 2021-03-09T10:10:44.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800488b0 con 0x7ff980050140 2021-03-09T10:10:44.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:44.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:44.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fc20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:44.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:44.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:44.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff98003fc20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:44.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff98003fc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:44.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:44.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:44.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035fb0 con 0x7ff980050140 2021-03-09T10:10:44.674 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:44.663+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13526 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:44.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.663+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013520 con 0x7ff980050140 2021-03-09T10:10:44.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.667+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 rx=0x7ff98c008a40 tx=0x7ff98c00bef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:44.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.667+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980050140 2021-03-09T10:10:44.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff980050140 2021-03-09T10:10:44.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.675+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980050140 2021-03-09T10:10:44.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.675+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:44.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.675+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:44.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:44.675+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:49.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:49.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:49.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 rx=0x7ff98c008a40 tx=0x7ff98c00bef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:49.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 rx=0x7ff98c008a40 tx=0x7ff98c00bef0).stop 2021-03-09T10:10:49.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:49.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=CLOSED pgs=4096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:49.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:49.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:49.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:49.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046570 con 0x7ff99c0adc20 2021-03-09T10:10:49.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005cd0 con 0x7ff99c0613a0 2021-03-09T10:10:49.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f140 con 0x7ff980051540 2021-03-09T10:10:49.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:49.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:49.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:49.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:49.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:49.676 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:49.667+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13568 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:49.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:49.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.667+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:49.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:49.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:49.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980095100 con 0x7ff980051540 2021-03-09T10:10:49.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027320 con 0x7ff980051540 2021-03-09T10:10:49.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 secure :-1 s=READY pgs=4110 cs=0 l=1 rev1=1 rx=0x7ff980050090 tx=0x7ff994011400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:49.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:10:49.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.675+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d910 con 0x7ff980051540 2021-03-09T10:10:49.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.675+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:10:49.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.675+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:49.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.679+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:49.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:49.679+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:54.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:54.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:54.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 secure :-1 s=READY pgs=4110 cs=0 l=1 rev1=1 rx=0x7ff980050090 tx=0x7ff994011400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:54.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 secure :-1 s=READY pgs=4110 cs=0 l=1 rev1=1 rx=0x7ff980050090 tx=0x7ff994011400).stop 2021-03-09T10:10:54.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:54.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094230 unknown :-1 s=CLOSED pgs=4110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:54.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:54.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:54.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:54.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051e00 con 0x7ff99c0adc20 2021-03-09T10:10:54.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ae40 con 0x7ff99c0613a0 2021-03-09T10:10:54.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005cd0 con 0x7ff980050140 2021-03-09T10:10:54.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:54.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:54.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:54.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:54.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:54.678 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:54.671+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13601 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:54.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:54.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:54.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:54.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:54.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980029120 con 0x7ff980050140 2021-03-09T10:10:54.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980095100 con 0x7ff980050140 2021-03-09T10:10:54.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4117 cs=0 l=1 rev1=1 rx=0x7ff994092240 tx=0x7ff994009d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:54.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409aec0 con 0x7ff980050140 2021-03-09T10:10:54.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.675+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994012df0 con 0x7ff980050140 2021-03-09T10:10:54.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:10:54.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.679+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:54.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.679+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:54.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:54.679+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:59.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:59.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:59.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4117 cs=0 l=1 rev1=1 rx=0x7ff994092240 tx=0x7ff994009d80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:59.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4117 cs=0 l=1 rev1=1 rx=0x7ff994092240 tx=0x7ff994009d80).stop 2021-03-09T10:10:59.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:59.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=CLOSED pgs=4117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:59.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:59.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:59.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800292b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:59.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f2e0 con 0x7ff99c0adc20 2021-03-09T10:10:59.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800303d0 con 0x7ff99c0613a0 2021-03-09T10:10:59.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ae40 con 0x7ff980051540 2021-03-09T10:10:59.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:10:59.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:59.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:10:59.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:59.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800292b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:59.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:59.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:59.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:59.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:59.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800952a0 con 0x7ff980051540 2021-03-09T10:10:59.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.671+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980029120 con 0x7ff980051540 2021-03-09T10:10:59.683 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:10:59.671+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13640 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:10:59.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.675+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800292b0 secure :-1 s=READY pgs=4128 cs=0 l=1 rev1=1 rx=0x7ff99408df90 tx=0x7ff99407a1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:59.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.675+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980051540 2021-03-09T10:10:59.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.675+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098ce0 con 0x7ff980051540 2021-03-09T10:10:59.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:10:59.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.679+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:59.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.679+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:59.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:10:59.679+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:10:59.735 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:10:59.738 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:11:04.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800292b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:04.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800292b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:04.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800292b0 secure :-1 s=READY pgs=4128 cs=0 l=1 rev1=1 rx=0x7ff99408df90 tx=0x7ff99407a1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:04.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800292b0 secure :-1 s=READY pgs=4128 cs=0 l=1 rev1=1 rx=0x7ff99408df90 tx=0x7ff99407a1e0).stop 2021-03-09T10:11:04.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800292b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:04.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800292b0 unknown :-1 s=CLOSED pgs=4128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:04.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:04.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:04.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:04.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ae40 con 0x7ff99c0adc20 2021-03-09T10:11:04.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800303d0 con 0x7ff99c0613a0 2021-03-09T10:11:04.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003ae80 con 0x7ff980050140 2021-03-09T10:11:04.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:04.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:04.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:04.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:04.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:04.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:04.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:04.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:04.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:04.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980029120 con 0x7ff980050140 2021-03-09T10:11:04.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800952a0 con 0x7ff980050140 2021-03-09T10:11:04.689 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:04.675+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13674 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:04.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.675+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4137 cs=0 l=1 rev1=1 rx=0x7ff980013a60 tx=0x7ff994029f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:04.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940970e0 con 0x7ff980050140 2021-03-09T10:11:04.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.683+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994029ce0 con 0x7ff980050140 2021-03-09T10:11:04.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.683+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:11:04.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.683+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:04.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.683+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:04.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:04.683+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:09.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:11:09.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:09.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4137 cs=0 l=1 rev1=1 rx=0x7ff980013a60 tx=0x7ff994029f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:09.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4137 cs=0 l=1 rev1=1 rx=0x7ff980013a60 tx=0x7ff994029f00).stop 2021-03-09T10:11:09.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:09.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=CLOSED pgs=4137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:09.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:09.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:09.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:09.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ae40 con 0x7ff99c0adc20 2021-03-09T10:11:09.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008a980 con 0x7ff99c0613a0 2021-03-09T10:11:09.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055310 con 0x7ff980051540 2021-03-09T10:11:09.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:09.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:09.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:09.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:09.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:09.686 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:09.679+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13709 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:09.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:09.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:09.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:09.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:09.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980095100 con 0x7ff980051540 2021-03-09T10:11:09.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035fb0 con 0x7ff980051540 2021-03-09T10:11:09.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.679+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094c10 secure :-1 s=READY pgs=4150 cs=0 l=1 rev1=1 rx=0x7ff98c008d20 tx=0x7ff98c008cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:09.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.683+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980051540 2021-03-09T10:11:09.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.683+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980051540 2021-03-09T10:11:09.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.687+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:11:09.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.687+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:09.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.687+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:09.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:09.687+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:14.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:14.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:14.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094c10 secure :-1 s=READY pgs=4150 cs=0 l=1 rev1=1 rx=0x7ff98c008d20 tx=0x7ff98c008cb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:14.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094c10 secure :-1 s=READY pgs=4150 cs=0 l=1 rev1=1 rx=0x7ff98c008d20 tx=0x7ff98c008cb0).stop 2021-03-09T10:11:14.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:14.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094c10 unknown :-1 s=CLOSED pgs=4150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:14.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:14.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:14.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:14.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d650 con 0x7ff99c0adc20 2021-03-09T10:11:14.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037ae0 con 0x7ff99c0613a0 2021-03-09T10:11:14.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035fb0 con 0x7ff980050140 2021-03-09T10:11:14.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:14.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:14.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:14.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:14.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:14.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:14.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:14.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:14.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:14.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004f120 con 0x7ff980050140 2021-03-09T10:11:14.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.679+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980029120 con 0x7ff980050140 2021-03-09T10:11:14.692 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:14.683+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13743 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:14.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.683+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4159 cs=0 l=1 rev1=1 rx=0x7ff980011350 tx=0x7ff99403ef10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:14.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.683+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994014120 con 0x7ff980050140 2021-03-09T10:11:14.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.683+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994084e20 con 0x7ff980050140 2021-03-09T10:11:14.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:11:14.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.691+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:14.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.691+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:14.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:14.691+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:19.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:19.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:19.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4159 cs=0 l=1 rev1=1 rx=0x7ff980011350 tx=0x7ff99403ef10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:19.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 secure :-1 s=READY pgs=4159 cs=0 l=1 rev1=1 rx=0x7ff980011350 tx=0x7ff99403ef10).stop 2021-03-09T10:11:19.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:19.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094720 unknown :-1 s=CLOSED pgs=4159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:19.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:19.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:19.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:19.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027280 con 0x7ff99c0adc20 2021-03-09T10:11:19.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094ed0 con 0x7ff99c0613a0 2021-03-09T10:11:19.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980028f80 con 0x7ff980051540 2021-03-09T10:11:19.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:19.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:19.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:19.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:19.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:19.693 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:19.683+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13769 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:19.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980094230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:19.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980094230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:19.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:19.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:19.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800290c0 con 0x7ff980051540 2021-03-09T10:11:19.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.683+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004f120 con 0x7ff980051540 2021-03-09T10:11:19.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.687+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a920 secure :-1 s=READY pgs=4169 cs=0 l=1 rev1=1 rx=0x7ff994071880 tx=0x7ff994014210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:19.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.687+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940700b0 con 0x7ff980051540 2021-03-09T10:11:19.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.687+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994070210 con 0x7ff980051540 2021-03-09T10:11:19.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:11:19.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.691+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:19.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.691+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:19.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:19.691+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:24.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:24.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:24.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a920 secure :-1 s=READY pgs=4169 cs=0 l=1 rev1=1 rx=0x7ff994071880 tx=0x7ff994014210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:24.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a920 secure :-1 s=READY pgs=4169 cs=0 l=1 rev1=1 rx=0x7ff994071880 tx=0x7ff994014210).stop 2021-03-09T10:11:24.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008a920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:24.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008a920 unknown :-1 s=CLOSED pgs=4169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:24.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:24.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:24.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:24.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005cd0 con 0x7ff99c0adc20 2021-03-09T10:11:24.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035fb0 con 0x7ff99c0613a0 2021-03-09T10:11:24.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037920 con 0x7ff980050140 2021-03-09T10:11:24.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:24.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:24.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:24.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:24.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:24.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:24.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:24.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:24.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:24.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980037a60 con 0x7ff980050140 2021-03-09T10:11:24.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800290c0 con 0x7ff980050140 2021-03-09T10:11:24.698 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:24.687+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13812 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:24.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.687+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 secure :-1 s=READY pgs=4180 cs=0 l=1 rev1=1 rx=0x7ff99405dba0 tx=0x7ff99407a180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:24.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405de40 con 0x7ff980050140 2021-03-09T10:11:24.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:11:24.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.695+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980050140 2021-03-09T10:11:24.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.695+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:24.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.695+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:24.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:24.695+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:29.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:29.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:29.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 secure :-1 s=READY pgs=4180 cs=0 l=1 rev1=1 rx=0x7ff99405dba0 tx=0x7ff99407a180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:29.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 secure :-1 s=READY pgs=4180 cs=0 l=1 rev1=1 rx=0x7ff99405dba0 tx=0x7ff99407a180).stop 2021-03-09T10:11:29.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:29.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 unknown :-1 s=CLOSED pgs=4180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:29.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:29.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:29.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:29.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005d20 con 0x7ff99c0adc20 2021-03-09T10:11:29.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800378d0 con 0x7ff99c0613a0 2021-03-09T10:11:29.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027210 con 0x7ff980051540 2021-03-09T10:11:29.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:29.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:29.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:29.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:29.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:29.698 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:29.691+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13855 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:29.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:29.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:29.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:29.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:29.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008a880 con 0x7ff980051540 2021-03-09T10:11:29.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980037a60 con 0x7ff980051540 2021-03-09T10:11:29.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094720 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994094130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:29.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.695+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409a0e0 con 0x7ff980051540 2021-03-09T10:11:29.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.695+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940718f0 con 0x7ff980051540 2021-03-09T10:11:29.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.699+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:11:29.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.699+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:29.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.699+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:29.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:29.699+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:29.782 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:11:29.784 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:11:34.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:34.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:34.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094720 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994094130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:34.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094720 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994094130).stop 2021-03-09T10:11:34.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980094720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:34.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980094720 unknown :-1 s=CLOSED pgs=4191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:34.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:34.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:34.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:34.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037a70 con 0x7ff99c0adc20 2021-03-09T10:11:34.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035fb0 con 0x7ff99c0613a0 2021-03-09T10:11:34.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008aa10 con 0x7ff980050140 2021-03-09T10:11:34.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:34.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:34.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:34.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:34.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.691+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:34.701 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:34.691+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13889 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:34.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:34.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.695+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:34.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:34.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.695+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:34.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008ab80 con 0x7ff980050140 2021-03-09T10:11:34.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008a880 con 0x7ff980050140 2021-03-09T10:11:34.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.695+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 secure :-1 s=READY pgs=4202 cs=0 l=1 rev1=1 rx=0x7ff99403e040 tx=0x7ff99402a3f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:34.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.699+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405d9e0 con 0x7ff980050140 2021-03-09T10:11:34.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.699+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405d9e0 con 0x7ff980050140 2021-03-09T10:11:34.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.703+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:11:34.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.703+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:34.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.707+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:34.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:34.707+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:39.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:39.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:39.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 secure :-1 s=READY pgs=4202 cs=0 l=1 rev1=1 rx=0x7ff99403e040 tx=0x7ff99402a3f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:39.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 secure :-1 s=READY pgs=4202 cs=0 l=1 rev1=1 rx=0x7ff99403e040 tx=0x7ff99402a3f0).stop 2021-03-09T10:11:39.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980094230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:39.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980094230 unknown :-1 s=CLOSED pgs=4202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:39.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:39.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:39.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:39.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056850 con 0x7ff99c0adc20 2021-03-09T10:11:39.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a200 con 0x7ff99c0613a0 2021-03-09T10:11:39.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037a70 con 0x7ff980051540 2021-03-09T10:11:39.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:39.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:39.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:39.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:39.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:39.704 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:39.695+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13924 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:39.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:39.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:39.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:39.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:39.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027210 con 0x7ff980051540 2021-03-09T10:11:39.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ab80 con 0x7ff980051540 2021-03-09T10:11:39.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.695+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004a990 secure :-1 s=READY pgs=4212 cs=0 l=1 rev1=1 rx=0x7ff9940864e0 tx=0x7ff994081340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:39.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.699+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009a20 con 0x7ff980051540 2021-03-09T10:11:39.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.699+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:11:39.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.703+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:11:39.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.703+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:39.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.707+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:39.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:39.707+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:44.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98004a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:44.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98004a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:44.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004a990 secure :-1 s=READY pgs=4212 cs=0 l=1 rev1=1 rx=0x7ff9940864e0 tx=0x7ff994081340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:44.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004a990 secure :-1 s=READY pgs=4212 cs=0 l=1 rev1=1 rx=0x7ff9940864e0 tx=0x7ff994081340).stop 2021-03-09T10:11:44.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98004a990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:44.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98004a990 unknown :-1 s=CLOSED pgs=4212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:44.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:44.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:44.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:44.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056850 con 0x7ff99c0adc20 2021-03-09T10:11:44.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008aa00 con 0x7ff99c0613a0 2021-03-09T10:11:44.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037a70 con 0x7ff980050140 2021-03-09T10:11:44.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:44.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:44.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:44.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:44.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:44.707 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:44.699+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13959 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:44.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:44.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:44.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:44.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:44.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027210 con 0x7ff980050140 2021-03-09T10:11:44.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980095100 con 0x7ff980050140 2021-03-09T10:11:44.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.699+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037d30 secure :-1 s=READY pgs=4226 cs=0 l=1 rev1=1 rx=0x7ff980006930 tx=0x7ff98c0a2f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:44.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.703+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:11:44.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.703+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c084e80 con 0x7ff980050140 2021-03-09T10:11:44.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.711+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980050140 2021-03-09T10:11:44.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.711+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:44.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.711+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:44.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:44.711+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:49.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:49.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:49.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037d30 secure :-1 s=READY pgs=4226 cs=0 l=1 rev1=1 rx=0x7ff980006930 tx=0x7ff98c0a2f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:49.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037d30 secure :-1 s=READY pgs=4226 cs=0 l=1 rev1=1 rx=0x7ff980006930 tx=0x7ff98c0a2f00).stop 2021-03-09T10:11:49.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:49.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037d30 unknown :-1 s=CLOSED pgs=4226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:49.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980038220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:49.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:49.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:49.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033480 con 0x7ff99c0adc20 2021-03-09T10:11:49.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014720 con 0x7ff99c0613a0 2021-03-09T10:11:49.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055bd0 con 0x7ff980051540 2021-03-09T10:11:49.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:49.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:49.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:49.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:49.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.698+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980038220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:49.710 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:49.702+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13999 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:49.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.702+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:49.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.702+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:49.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.702+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:49.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.702+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:49.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.702+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027210 con 0x7ff980051540 2021-03-09T10:11:49.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.702+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ab80 con 0x7ff980051540 2021-03-09T10:11:49.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980038220 secure :-1 s=READY pgs=4235 cs=0 l=1 rev1=1 rx=0x7ff994093480 tx=0x7ff99407dda0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:49.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.706+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994075eb0 con 0x7ff980051540 2021-03-09T10:11:49.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994075eb0 con 0x7ff980051540 2021-03-09T10:11:49.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:11:49.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.710+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:49.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.710+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:49.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:49.710+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:54.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980038220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:54.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980038220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:54.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980038220 secure :-1 s=READY pgs=4235 cs=0 l=1 rev1=1 rx=0x7ff994093480 tx=0x7ff99407dda0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:54.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980038220 secure :-1 s=READY pgs=4235 cs=0 l=1 rev1=1 rx=0x7ff994093480 tx=0x7ff99407dda0).stop 2021-03-09T10:11:54.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980038220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:54.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980038220 unknown :-1 s=CLOSED pgs=4235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:54.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800143e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:54.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:54.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:54.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003aa0 con 0x7ff99c0adc20 2021-03-09T10:11:54.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a980 con 0x7ff99c0613a0 2021-03-09T10:11:54.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033480 con 0x7ff980050140 2021-03-09T10:11:54.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:54.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:54.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:54.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:54.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.702+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800143e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:54.714 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:54.702+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14030 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:54.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.706+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:54.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:54.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.706+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:54.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:54.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.706+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027210 con 0x7ff980050140 2021-03-09T10:11:54.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.706+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ab80 con 0x7ff980050140 2021-03-09T10:11:54.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800143e0 secure :-1 s=READY pgs=4246 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99408b880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:54.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980050140 2021-03-09T10:11:54.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99403ea60 con 0x7ff980050140 2021-03-09T10:11:54.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.714+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:11:54.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.714+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:54.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.714+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:54.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:54.714+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:59.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800143e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:59.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800143e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:59.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800143e0 secure :-1 s=READY pgs=4246 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99408b880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:59.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800143e0 secure :-1 s=READY pgs=4246 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99408b880).stop 2021-03-09T10:11:59.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800143e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:59.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800143e0 unknown :-1 s=CLOSED pgs=4246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:59.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:59.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:59.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:59.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003aa0 con 0x7ff99c0adc20 2021-03-09T10:11:59.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ab20 con 0x7ff99c0613a0 2021-03-09T10:11:59.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003d40 con 0x7ff980051540 2021-03-09T10:11:59.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:11:59.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:11:59.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:59.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:59.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:59.716 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:11:59.706+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14058 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:11:59.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:59.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:59.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.710+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:59.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.710+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:59.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.710+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027210 con 0x7ff980051540 2021-03-09T10:11:59.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.710+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ab80 con 0x7ff980051540 2021-03-09T10:11:59.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.710+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002d0c0 secure :-1 s=READY pgs=4257 cs=0 l=1 rev1=1 rx=0x7ff99408ba40 tx=0x7ff9940aa080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:59.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980051540 2021-03-09T10:11:59.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409aec0 con 0x7ff980051540 2021-03-09T10:11:59.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:11:59.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.718+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:59.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.718+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:59.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:11:59.718+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:11:59.828 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:11:59.831 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:12:04.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.706+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:04.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.706+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:04.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002d0c0 secure :-1 s=READY pgs=4257 cs=0 l=1 rev1=1 rx=0x7ff99408ba40 tx=0x7ff9940aa080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:04.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.706+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002d0c0 secure :-1 s=READY pgs=4257 cs=0 l=1 rev1=1 rx=0x7ff99408ba40 tx=0x7ff9940aa080).stop 2021-03-09T10:12:04.716 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:04.710+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14103 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:04.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.706+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98002d0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:04.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98002d0c0 unknown :-1 s=CLOSED pgs=4257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:04.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:04.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:04.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:04.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003aa0 con 0x7ff99c0adc20 2021-03-09T10:12:04.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ab20 con 0x7ff99c0613a0 2021-03-09T10:12:04.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800956a0 con 0x7ff980050140 2021-03-09T10:12:04.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:04.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:04.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:04.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:04.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.710+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:04.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.714+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:04.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.714+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:04.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.714+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:04.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.714+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:04.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.714+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027210 con 0x7ff980050140 2021-03-09T10:12:04.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.714+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980095100 con 0x7ff980050140 2021-03-09T10:12:04.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.714+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 secure :-1 s=READY pgs=4271 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00baf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:04.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.714+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:12:04.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980050140 2021-03-09T10:12:04.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c008d20 con 0x7ff980050140 2021-03-09T10:12:04.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.722+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:04.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.722+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:04.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:04.722+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:09.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800381b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:09.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800381b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:09.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 secure :-1 s=READY pgs=4271 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00baf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:09.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 secure :-1 s=READY pgs=4271 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00baf0).stop 2021-03-09T10:12:09.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800381b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:09.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 unknown :-1 s=CLOSED pgs=4271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:09.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:09.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:09.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:09.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:09.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:09.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003aa0 con 0x7ff99c0adc20 2021-03-09T10:12:09.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005e30 con 0x7ff99c0613a0 2021-03-09T10:12:09.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013ae0 con 0x7ff980051540 2021-03-09T10:12:09.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:09.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:09.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.710+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:09.722 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:09.714+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14142 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:09.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.714+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:09.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.714+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:09.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.714+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:09.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.714+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:09.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.714+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980010fc0 con 0x7ff980051540 2021-03-09T10:12:09.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.714+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ab80 con 0x7ff980051540 2021-03-09T10:12:09.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091fc0 secure :-1 s=READY pgs=4283 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99401cac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:09.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980051540 2021-03-09T10:12:09.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940930e0 con 0x7ff980051540 2021-03-09T10:12:09.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:12:09.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.722+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:09.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.722+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:09.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:09.722+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:14.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:14.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:14.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091fc0 secure :-1 s=READY pgs=4283 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99401cac0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:14.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091fc0 secure :-1 s=READY pgs=4283 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99401cac0).stop 2021-03-09T10:12:14.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:14.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091fc0 unknown :-1 s=CLOSED pgs=4283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:14.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:14.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:14.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:14.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005e30 con 0x7ff99c0adc20 2021-03-09T10:12:14.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003aa0 con 0x7ff99c0613a0 2021-03-09T10:12:14.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013ae0 con 0x7ff980050140 2021-03-09T10:12:14.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:14.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:14.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:14.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:14.727 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:14.718+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14174 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:14.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:14.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:14.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:14.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:14.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:14.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054fe0 con 0x7ff980050140 2021-03-09T10:12:14.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.718+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980010fc0 con 0x7ff980050140 2021-03-09T10:12:14.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.722+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 secure :-1 s=READY pgs=4295 cs=0 l=1 rev1=1 rx=0x7ff980006930 tx=0x7ff99409b200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:14.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.726+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069030 con 0x7ff980050140 2021-03-09T10:12:14.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.726+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994064090 con 0x7ff980050140 2021-03-09T10:12:14.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:12:14.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.730+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:14.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.730+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:14.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:14.730+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:19.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800381b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:19.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800381b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:19.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 secure :-1 s=READY pgs=4295 cs=0 l=1 rev1=1 rx=0x7ff980006930 tx=0x7ff99409b200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:19.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 secure :-1 s=READY pgs=4295 cs=0 l=1 rev1=1 rx=0x7ff980006930 tx=0x7ff99409b200).stop 2021-03-09T10:12:19.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800381b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:19.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 unknown :-1 s=CLOSED pgs=4295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:19.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:19.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:19.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:19.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f2c0 con 0x7ff99c0adc20 2021-03-09T10:12:19.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003aa0 con 0x7ff99c0613a0 2021-03-09T10:12:19.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005e30 con 0x7ff980051540 2021-03-09T10:12:19.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:19.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:19.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:19.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:19.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.718+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:19.729 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:19.718+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14212 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:19.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.722+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:19.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.722+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:19.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.722+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:19.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.722+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:19.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.722+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800360f0 con 0x7ff980051540 2021-03-09T10:12:19.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.722+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054fe0 con 0x7ff980051540 2021-03-09T10:12:19.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.722+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=4301 cs=0 l=1 rev1=1 rx=0x7ff99407a230 tx=0x7ff9940831a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:19.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994064090 con 0x7ff980051540 2021-03-09T10:12:19.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.726+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994064090 con 0x7ff980051540 2021-03-09T10:12:19.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:12:19.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.730+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:19.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.730+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:19.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:19.730+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:24.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:24.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:24.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=4301 cs=0 l=1 rev1=1 rx=0x7ff99407a230 tx=0x7ff9940831a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:24.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=4301 cs=0 l=1 rev1=1 rx=0x7ff99407a230 tx=0x7ff9940831a0).stop 2021-03-09T10:12:24.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:24.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=CLOSED pgs=4301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:24.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:24.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:24.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:24.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003aa0 con 0x7ff99c0adc20 2021-03-09T10:12:24.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005e30 con 0x7ff99c0613a0 2021-03-09T10:12:24.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013fa0 con 0x7ff980050140 2021-03-09T10:12:24.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:24.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:24.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:24.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:24.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.722+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:24.733 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:24.722+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14247 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:24.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.726+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:24.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.726+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:24.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.726+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:24.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.726+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:24.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.726+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088cc0 con 0x7ff980050140 2021-03-09T10:12:24.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.726+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980095100 con 0x7ff980050140 2021-03-09T10:12:24.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.730+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 secure :-1 s=READY pgs=4316 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c003ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:24.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:12:24.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.734+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980050140 2021-03-09T10:12:24.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.734+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c008bd0 con 0x7ff980050140 2021-03-09T10:12:24.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.734+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:24.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.734+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:24.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:24.734+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:29.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800381b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:29.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800381b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:29.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 secure :-1 s=READY pgs=4316 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c003ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:29.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 secure :-1 s=READY pgs=4316 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c003ef0).stop 2021-03-09T10:12:29.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800381b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:29.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800381b0 unknown :-1 s=CLOSED pgs=4316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:29.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:29.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:29.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:29.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003d40 con 0x7ff99c0adc20 2021-03-09T10:12:29.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008aba0 con 0x7ff99c0613a0 2021-03-09T10:12:29.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800271a0 con 0x7ff980051540 2021-03-09T10:12:29.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:29.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:29.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:29.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.722+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:29.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.726+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:29.735 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:29.726+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14286 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:29.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.726+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:29.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.726+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:29.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.726+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:29.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.726+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:29.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.726+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054fe0 con 0x7ff980051540 2021-03-09T10:12:29.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.726+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800360f0 con 0x7ff980051540 2021-03-09T10:12:29.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.730+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000ab20 secure :-1 s=READY pgs=4323 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff994078000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:29.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980051540 2021-03-09T10:12:29.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994014120 con 0x7ff980051540 2021-03-09T10:12:29.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.734+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:12:29.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.734+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:29.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.734+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:29.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:29.734+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:29.885 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:12:29.888 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:12:34.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:34.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:34.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000ab20 secure :-1 s=READY pgs=4323 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff994078000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:34.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000ab20 secure :-1 s=READY pgs=4323 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff994078000).stop 2021-03-09T10:12:34.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000ab20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:34.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000ab20 unknown :-1 s=CLOSED pgs=4323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:34.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:34.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:34.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:34.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008aba0 con 0x7ff99c0adc20 2021-03-09T10:12:34.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d8e0 con 0x7ff99c0613a0 2021-03-09T10:12:34.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003cf0 con 0x7ff980050140 2021-03-09T10:12:34.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:34.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:34.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:34.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:34.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:34.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:34.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:34.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:34.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:34.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035f50 con 0x7ff980050140 2021-03-09T10:12:34.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.726+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088cc0 con 0x7ff980050140 2021-03-09T10:12:34.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.730+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 secure :-1 s=READY pgs=4329 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c0469b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:34.739 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:34.730+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14323 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:34.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:12:34.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c009540 con 0x7ff980050140 2021-03-09T10:12:34.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.738+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980050140 2021-03-09T10:12:34.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.738+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:34.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.738+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:34.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:34.738+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:39.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98003d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:39.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98003d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:39.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 secure :-1 s=READY pgs=4329 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c0469b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:39.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 secure :-1 s=READY pgs=4329 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c0469b0).stop 2021-03-09T10:12:39.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98003d3f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:39.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 unknown :-1 s=CLOSED pgs=4329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:39.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:39.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:39.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:39.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008aba0 con 0x7ff99c0adc20 2021-03-09T10:12:39.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003da80 con 0x7ff99c0613a0 2021-03-09T10:12:39.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800271a0 con 0x7ff980051540 2021-03-09T10:12:39.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:39.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:39.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:39.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:39.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.730+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:39.741 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:39.730+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14363 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:39.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.734+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:39.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.734+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:39.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.734+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:39.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.734+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:39.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.734+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000ae70 con 0x7ff980051540 2021-03-09T10:12:39.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.734+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035f50 con 0x7ff980051540 2021-03-09T10:12:39.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.734+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 secure :-1 s=READY pgs=4344 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c07dcb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:39.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.738+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04bea0 con 0x7ff980051540 2021-03-09T10:12:39.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.738+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04bea0 con 0x7ff980051540 2021-03-09T10:12:39.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.742+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00b120 con 0x7ff980051540 2021-03-09T10:12:39.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.742+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:39.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.742+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:39.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:39.742+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:44.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:44.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:44.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 secure :-1 s=READY pgs=4344 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c07dcb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:44.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 secure :-1 s=READY pgs=4344 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c07dcb0).stop 2021-03-09T10:12:44.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000a980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:44.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 unknown :-1 s=CLOSED pgs=4344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:44.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:44.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:44.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:44.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019260 con 0x7ff99c0adc20 2021-03-09T10:12:44.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980031410 con 0x7ff99c0613a0 2021-03-09T10:12:44.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d8e0 con 0x7ff980050140 2021-03-09T10:12:44.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:44.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:44.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:44.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:44.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:44.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:44.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:44.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:44.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:44.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003da20 con 0x7ff980050140 2021-03-09T10:12:44.745 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:44.734+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14397 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:44.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000ae70 con 0x7ff980050140 2021-03-09T10:12:44.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.734+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 secure :-1 s=READY pgs=4352 cs=0 l=1 rev1=1 rx=0x7ff98c074f60 tx=0x7ff98c00bf60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:44.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.738+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c052130 con 0x7ff980050140 2021-03-09T10:12:44.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.738+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08f360 con 0x7ff980050140 2021-03-09T10:12:44.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.742+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c08ed00 con 0x7ff980050140 2021-03-09T10:12:44.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.742+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:44.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.742+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:44.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:44.742+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:49.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98003d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:49.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98003d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:49.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 secure :-1 s=READY pgs=4352 cs=0 l=1 rev1=1 rx=0x7ff98c074f60 tx=0x7ff98c00bf60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:49.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 secure :-1 s=READY pgs=4352 cs=0 l=1 rev1=1 rx=0x7ff98c074f60 tx=0x7ff98c00bf60).stop 2021-03-09T10:12:49.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98003d3f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:49.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 unknown :-1 s=CLOSED pgs=4352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:49.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:49.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:49.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:49.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019210 con 0x7ff99c0adc20 2021-03-09T10:12:49.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800271a0 con 0x7ff99c0613a0 2021-03-09T10:12:49.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ae70 con 0x7ff980051540 2021-03-09T10:12:49.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:49.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:49.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:49.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:49.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:49.747 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:49.738+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14437 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:49.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:49.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:49.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:49.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:49.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000afe0 con 0x7ff980051540 2021-03-09T10:12:49.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054fe0 con 0x7ff980051540 2021-03-09T10:12:49.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 secure :-1 s=READY pgs=4361 cs=0 l=1 rev1=1 rx=0x7ff98003d150 tx=0x7ff99406ddb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:49.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.738+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405fe60 con 0x7ff980051540 2021-03-09T10:12:49.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.742+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405fe60 con 0x7ff980051540 2021-03-09T10:12:49.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.746+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:12:49.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.746+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:49.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.746+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:49.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:49.746+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:54.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:54.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:54.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 secure :-1 s=READY pgs=4361 cs=0 l=1 rev1=1 rx=0x7ff98003d150 tx=0x7ff99406ddb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:54.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 secure :-1 s=READY pgs=4361 cs=0 l=1 rev1=1 rx=0x7ff98003d150 tx=0x7ff99406ddb0).stop 2021-03-09T10:12:54.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000a980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:54.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 unknown :-1 s=CLOSED pgs=4361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:54.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:54.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:54.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:54.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093e60 con 0x7ff99c0adc20 2021-03-09T10:12:54.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ae70 con 0x7ff99c0613a0 2021-03-09T10:12:54.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054e40 con 0x7ff980050140 2021-03-09T10:12:54.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:54.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:54.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:54.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:54.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:54.750 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:54.742+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14468 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:54.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:54.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:54.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:54.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:54.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054f80 con 0x7ff980050140 2021-03-09T10:12:54.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003da20 con 0x7ff980050140 2021-03-09T10:12:54.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.742+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 secure :-1 s=READY pgs=4374 cs=0 l=1 rev1=1 rx=0x7ff98c005620 tx=0x7ff98c0a0130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:54.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.746+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c005650 con 0x7ff980050140 2021-03-09T10:12:54.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.750+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c005650 con 0x7ff980050140 2021-03-09T10:12:54.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980050140 2021-03-09T10:12:54.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.754+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:54.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.754+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:54.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:54.754+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:59.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98003d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:59.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98003d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:59.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 secure :-1 s=READY pgs=4374 cs=0 l=1 rev1=1 rx=0x7ff98c005620 tx=0x7ff98c0a0130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:59.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 secure :-1 s=READY pgs=4374 cs=0 l=1 rev1=1 rx=0x7ff98c005620 tx=0x7ff98c0a0130).stop 2021-03-09T10:12:59.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98003d3f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:59.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98003d3f0 unknown :-1 s=CLOSED pgs=4374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:59.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:59.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:59.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:59.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054e40 con 0x7ff99c0adc20 2021-03-09T10:12:59.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ae70 con 0x7ff99c0613a0 2021-03-09T10:12:59.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800271a0 con 0x7ff980051540 2021-03-09T10:12:59.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:12:59.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:59.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:12:59.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:59.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:59.753 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:12:59.746+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14511 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:12:59.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:59.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:59.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:59.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:59.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035f50 con 0x7ff980051540 2021-03-09T10:12:59.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000afe0 con 0x7ff980051540 2021-03-09T10:12:59.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.750+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 secure :-1 s=READY pgs=4388 cs=0 l=1 rev1=1 rx=0x7ff9940313b0 tx=0x7ff994078000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:59.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:12:59.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994092e50 con 0x7ff980051540 2021-03-09T10:12:59.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:12:59.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.754+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:59.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.754+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:59.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:12:59.754+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:12:59.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:12:59.960 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:13:04.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:13:04.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:04.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 secure :-1 s=READY pgs=4388 cs=0 l=1 rev1=1 rx=0x7ff9940313b0 tx=0x7ff994078000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:04.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 secure :-1 s=READY pgs=4388 cs=0 l=1 rev1=1 rx=0x7ff9940313b0 tx=0x7ff994078000).stop 2021-03-09T10:13:04.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000a980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:04.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000a980 unknown :-1 s=CLOSED pgs=4388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:04.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:04.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:04.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:04.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001b0c0 con 0x7ff99c0adc20 2021-03-09T10:13:04.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c370 con 0x7ff99c0613a0 2021-03-09T10:13:04.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d3c0 con 0x7ff980050140 2021-03-09T10:13:04.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:04.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:04.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:04.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:04.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:04.756 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:04.746+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14545 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:04.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:04.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:04.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:04.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:04.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003d530 con 0x7ff980050140 2021-03-09T10:13:04.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035f50 con 0x7ff980050140 2021-03-09T10:13:04.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.746+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000afb0 secure :-1 s=READY pgs=4394 cs=0 l=1 rev1=1 rx=0x7ff98004a8b0 tx=0x7ff99403bb80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:04.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.750+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980050140 2021-03-09T10:13:04.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.750+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:13:04.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:13:04.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.754+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:04.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.758+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:04.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:04.758+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:09.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:09.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:09.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000afb0 secure :-1 s=READY pgs=4394 cs=0 l=1 rev1=1 rx=0x7ff98004a8b0 tx=0x7ff99403bb80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:09.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000afb0 secure :-1 s=READY pgs=4394 cs=0 l=1 rev1=1 rx=0x7ff98004a8b0 tx=0x7ff99403bb80).stop 2021-03-09T10:13:09.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000afb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:09.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000afb0 unknown :-1 s=CLOSED pgs=4394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:09.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:09.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:09.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:09.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800271a0 con 0x7ff99c0adc20 2021-03-09T10:13:09.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a980 con 0x7ff99c0613a0 2021-03-09T10:13:09.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000aac0 con 0x7ff980051540 2021-03-09T10:13:09.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:09.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:09.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:09.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:09.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:09.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:09.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:09.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:09.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:09.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000ac00 con 0x7ff980051540 2021-03-09T10:13:09.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054f80 con 0x7ff980051540 2021-03-09T10:13:09.761 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:09.750+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14584 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:09.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.750+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800520d0 secure :-1 s=READY pgs=4404 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c016d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:09.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08f130 con 0x7ff980051540 2021-03-09T10:13:09.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c004070 con 0x7ff980051540 2021-03-09T10:13:09.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.758+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c046da0 con 0x7ff980051540 2021-03-09T10:13:09.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.758+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:09.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.758+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:09.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:09.758+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:14.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:13:14.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:14.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800520d0 secure :-1 s=READY pgs=4404 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c016d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:14.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800520d0 secure :-1 s=READY pgs=4404 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c016d60).stop 2021-03-09T10:13:14.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800520d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:14.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800520d0 unknown :-1 s=CLOSED pgs=4404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:14.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:14.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:14.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:14.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800525c0 con 0x7ff99c0adc20 2021-03-09T10:13:14.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035f50 con 0x7ff99c0613a0 2021-03-09T10:13:14.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff980050140 2021-03-09T10:13:14.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:14.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:14.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:14.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:14.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:14.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:14.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:14.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:14.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:14.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000b3c0 con 0x7ff980050140 2021-03-09T10:13:14.764 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:14.754+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14619 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:14.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003d530 con 0x7ff980050140 2021-03-09T10:13:14.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.754+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ad90 secure :-1 s=READY pgs=4415 cs=0 l=1 rev1=1 rx=0x7ff98008ce70 tx=0x7ff99400bed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:14.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.758+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980050140 2021-03-09T10:13:14.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.758+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994006bf0 con 0x7ff980050140 2021-03-09T10:13:14.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.762+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:13:14.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.762+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:14.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.762+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:14.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:14.762+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:19.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:19.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:19.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ad90 secure :-1 s=READY pgs=4415 cs=0 l=1 rev1=1 rx=0x7ff98008ce70 tx=0x7ff99400bed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:19.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ad90 secure :-1 s=READY pgs=4415 cs=0 l=1 rev1=1 rx=0x7ff98008ce70 tx=0x7ff99400bed0).stop 2021-03-09T10:13:19.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000ad90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:19.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ad90 unknown :-1 s=CLOSED pgs=4415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:19.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:19.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:19.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:19.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:19.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:19.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d6d0 con 0x7ff99c0adc20 2021-03-09T10:13:19.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800271a0 con 0x7ff99c0613a0 2021-03-09T10:13:19.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052520 con 0x7ff980051540 2021-03-09T10:13:19.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:19.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:19.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:19.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:19.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:19.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:19.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:19.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091e20 con 0x7ff980051540 2021-03-09T10:13:19.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000b3c0 con 0x7ff980051540 2021-03-09T10:13:19.768 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:19.757+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14659 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:19.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.757+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052030 secure :-1 s=READY pgs=4425 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff994006c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:19.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994096070 con 0x7ff980051540 2021-03-09T10:13:19.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:13:19.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.765+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:13:19.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.765+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:19.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.765+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:19.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:19.765+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:24.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980052030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:24.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980052030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:24.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052030 secure :-1 s=READY pgs=4425 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff994006c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:24.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052030 secure :-1 s=READY pgs=4425 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff994006c60).stop 2021-03-09T10:13:24.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980052030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:24.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980052030 unknown :-1 s=CLOSED pgs=4425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:24.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:24.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:24.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:24.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800889e0 con 0x7ff99c0adc20 2021-03-09T10:13:24.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800061b0 con 0x7ff99c0613a0 2021-03-09T10:13:24.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d6d0 con 0x7ff980050140 2021-03-09T10:13:24.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:24.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:24.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:24.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:24.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:24.769 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:24.761+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14690 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:24.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:24.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:24.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:24.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:24.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035f10 con 0x7ff980050140 2021-03-09T10:13:24.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091e20 con 0x7ff980050140 2021-03-09T10:13:24.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000b220 secure :-1 s=READY pgs=4440 cs=0 l=1 rev1=1 rx=0x7ff980009510 tx=0x7ff99403c8a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:24.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980050140 2021-03-09T10:13:24.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.765+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:13:24.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.769+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:13:24.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.769+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:24.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.769+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:24.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:24.769+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:29.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:29.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:29.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000b220 secure :-1 s=READY pgs=4440 cs=0 l=1 rev1=1 rx=0x7ff980009510 tx=0x7ff99403c8a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:29.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000b220 secure :-1 s=READY pgs=4440 cs=0 l=1 rev1=1 rx=0x7ff980009510 tx=0x7ff99403c8a0).stop 2021-03-09T10:13:29.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000b220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:29.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000b220 unknown :-1 s=CLOSED pgs=4440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:29.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:29.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:29.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:29.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090870 con 0x7ff99c0adc20 2021-03-09T10:13:29.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091010 con 0x7ff99c0613a0 2021-03-09T10:13:29.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d6d0 con 0x7ff980051540 2021-03-09T10:13:29.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:29.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:29.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:29.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:29.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:29.772 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:29.761+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14725 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:29.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:29.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:29.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:29.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:29.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800271a0 con 0x7ff980051540 2021-03-09T10:13:29.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.761+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035f10 con 0x7ff980051540 2021-03-09T10:13:29.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.765+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 secure :-1 s=READY pgs=4448 cs=0 l=1 rev1=1 rx=0x7ff9940667c0 tx=0x7ff994009d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:29.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.765+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99402a910 con 0x7ff980051540 2021-03-09T10:13:29.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.769+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99403e0d0 con 0x7ff980051540 2021-03-09T10:13:29.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.773+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:13:29.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.773+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:29.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.773+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:29.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:29.773+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:30.014 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:13:30.017 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:13:34.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:34.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:34.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 secure :-1 s=READY pgs=4448 cs=0 l=1 rev1=1 rx=0x7ff9940667c0 tx=0x7ff994009d80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:34.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 secure :-1 s=READY pgs=4448 cs=0 l=1 rev1=1 rx=0x7ff9940667c0 tx=0x7ff994009d80).stop 2021-03-09T10:13:34.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:34.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051fc0 unknown :-1 s=CLOSED pgs=4448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:34.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:34.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:34.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:34.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800911b0 con 0x7ff99c0adc20 2021-03-09T10:13:34.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090870 con 0x7ff99c0613a0 2021-03-09T10:13:34.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006890 con 0x7ff980050140 2021-03-09T10:13:34.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:34.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:34.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:34.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:34.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:34.775 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:34.765+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14767 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:34.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:34.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:34.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:34.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:34.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003d6d0 con 0x7ff980050140 2021-03-09T10:13:34.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035f10 con 0x7ff980050140 2021-03-09T10:13:34.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.765+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 secure :-1 s=READY pgs=4458 cs=0 l=1 rev1=1 rx=0x7ff980055a50 tx=0x7ff99407a2a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:34.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.769+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940826e0 con 0x7ff980050140 2021-03-09T10:13:34.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.769+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980050140 2021-03-09T10:13:34.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.773+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980050140 2021-03-09T10:13:34.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.773+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:34.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.773+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:34.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:34.773+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:39.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:39.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:39.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 secure :-1 s=READY pgs=4458 cs=0 l=1 rev1=1 rx=0x7ff980055a50 tx=0x7ff99407a2a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:39.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 secure :-1 s=READY pgs=4458 cs=0 l=1 rev1=1 rx=0x7ff980055a50 tx=0x7ff99407a2a0).stop 2021-03-09T10:13:39.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:39.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 unknown :-1 s=CLOSED pgs=4458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:39.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:39.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:39.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:39.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:39.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091350 con 0x7ff99c0adc20 2021-03-09T10:13:39.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:39.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:39.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:39.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008acb0 con 0x7ff99c0613a0 2021-03-09T10:13:39.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800271a0 con 0x7ff980051540 2021-03-09T10:13:39.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:39.778 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:39.769+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14807 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:39.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:39.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:39.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:39.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:39.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800272e0 con 0x7ff980051540 2021-03-09T10:13:39.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003d6d0 con 0x7ff980051540 2021-03-09T10:13:39.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.769+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 secure :-1 s=READY pgs=4469 cs=0 l=1 rev1=1 rx=0x7ff99409b350 tx=0x7ff994066730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:39.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.773+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405ed60 con 0x7ff980051540 2021-03-09T10:13:39.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.777+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99403e0d0 con 0x7ff980051540 2021-03-09T10:13:39.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.777+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:13:39.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.777+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:39.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.781+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:39.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:39.781+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:44.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:44.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:44.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 secure :-1 s=READY pgs=4469 cs=0 l=1 rev1=1 rx=0x7ff99409b350 tx=0x7ff994066730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:44.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 secure :-1 s=READY pgs=4469 cs=0 l=1 rev1=1 rx=0x7ff99409b350 tx=0x7ff994066730).stop 2021-03-09T10:13:44.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:44.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 unknown :-1 s=CLOSED pgs=4469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:44.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:44.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:44.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:44.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035eb0 con 0x7ff99c0adc20 2021-03-09T10:13:44.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091e20 con 0x7ff99c0613a0 2021-03-09T10:13:44.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091f60 con 0x7ff980050140 2021-03-09T10:13:44.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:44.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:44.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:44.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:44.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:44.781 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:44.773+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14841 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:44.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:44.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:44.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:44.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.773+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:44.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800920a0 con 0x7ff980050140 2021-03-09T10:13:44.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800272e0 con 0x7ff980050140 2021-03-09T10:13:44.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.777+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006550 secure :-1 s=READY pgs=4479 cs=0 l=1 rev1=1 rx=0x7ff980032a50 tx=0x7ff99401ddc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:44.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.781+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405ed60 con 0x7ff980050140 2021-03-09T10:13:44.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.781+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405ed60 con 0x7ff980050140 2021-03-09T10:13:44.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.785+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:13:44.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.785+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:44.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.785+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:44.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:44.785+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:49.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980006550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:49.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980006550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:49.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006550 secure :-1 s=READY pgs=4479 cs=0 l=1 rev1=1 rx=0x7ff980032a50 tx=0x7ff99401ddc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:49.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006550 secure :-1 s=READY pgs=4479 cs=0 l=1 rev1=1 rx=0x7ff980032a50 tx=0x7ff99401ddc0).stop 2021-03-09T10:13:49.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980006550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:49.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006550 unknown :-1 s=CLOSED pgs=4479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:49.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:49.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:49.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:49.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800271a0 con 0x7ff99c0adc20 2021-03-09T10:13:49.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800272e0 con 0x7ff99c0613a0 2021-03-09T10:13:49.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009070 con 0x7ff980051540 2021-03-09T10:13:49.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:49.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:49.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:49.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:49.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:49.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:49.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:49.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:49.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:49.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800091b0 con 0x7ff980051540 2021-03-09T10:13:49.784 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:49.777+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14880 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:49.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800920a0 con 0x7ff980051540 2021-03-09T10:13:49.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.781+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 secure :-1 s=READY pgs=4489 cs=0 l=1 rev1=1 rx=0x7ff994016580 tx=0x7ff99403ef50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:49.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.781+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff980051540 2021-03-09T10:13:49.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.785+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff980051540 2021-03-09T10:13:49.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:13:49.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.789+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:49.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.789+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:49.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:49.789+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:54.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:54.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:54.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 secure :-1 s=READY pgs=4489 cs=0 l=1 rev1=1 rx=0x7ff994016580 tx=0x7ff99403ef50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:54.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 secure :-1 s=READY pgs=4489 cs=0 l=1 rev1=1 rx=0x7ff994016580 tx=0x7ff99403ef50).stop 2021-03-09T10:13:54.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:54.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 unknown :-1 s=CLOSED pgs=4489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:54.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:54.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:54.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:54.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006850 con 0x7ff99c0adc20 2021-03-09T10:13:54.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0613a0 2021-03-09T10:13:54.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035f00 con 0x7ff980050140 2021-03-09T10:13:54.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:54.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:54.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:54.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.777+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:54.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.781+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:54.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.781+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:54.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.781+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:54.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.781+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:54.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.781+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:54.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.781+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091e20 con 0x7ff980050140 2021-03-09T10:13:54.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.781+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000ac00 con 0x7ff980050140 2021-03-09T10:13:54.788 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:54.781+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14915 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:54.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.781+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006360 secure :-1 s=READY pgs=4501 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c003ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:54.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:13:54.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.793+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980050140 2021-03-09T10:13:54.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.793+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c046d60 con 0x7ff980050140 2021-03-09T10:13:54.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.793+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:54.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.797+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:54.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:54.797+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:13:59.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980006360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:59.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980006360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:59.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006360 secure :-1 s=READY pgs=4501 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c003ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:59.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006360 secure :-1 s=READY pgs=4501 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c003ef0).stop 2021-03-09T10:13:59.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980006360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:59.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006360 unknown :-1 s=CLOSED pgs=4501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:59.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:59.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:59.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:59.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a7f0 con 0x7ff99c0adc20 2021-03-09T10:13:59.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003bb50 con 0x7ff99c0613a0 2021-03-09T10:13:59.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035f60 con 0x7ff980051540 2021-03-09T10:13:59.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:13:59.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:59.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:13:59.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:59.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.781+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:59.790 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:13:59.785+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14951 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:13:59.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.785+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:59.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.785+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:59.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.785+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:59.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.785+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:59.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.785+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000ada0 con 0x7ff980051540 2021-03-09T10:13:59.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.785+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800091b0 con 0x7ff980051540 2021-03-09T10:13:59.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.785+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 secure :-1 s=READY pgs=4513 cs=0 l=1 rev1=1 rx=0x7ff994034e50 tx=0x7ff994009dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:59.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020220 con 0x7ff980051540 2021-03-09T10:13:59.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994094130 con 0x7ff980051540 2021-03-09T10:13:59.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.793+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:13:59.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.793+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:59.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.793+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:59.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:13:59.793+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:00.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:14:00.061 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:14:04.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:04.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:04.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 secure :-1 s=READY pgs=4513 cs=0 l=1 rev1=1 rx=0x7ff994034e50 tx=0x7ff994009dc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:04.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 secure :-1 s=READY pgs=4513 cs=0 l=1 rev1=1 rx=0x7ff994034e50 tx=0x7ff994009dc0).stop 2021-03-09T10:14:04.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980051f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:04.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980051f90 unknown :-1 s=CLOSED pgs=4513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:04.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:04.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:04.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:04.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800067f0 con 0x7ff99c0adc20 2021-03-09T10:14:04.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0613a0 2021-03-09T10:14:04.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035f00 con 0x7ff980050140 2021-03-09T10:14:04.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:04.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:04.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:04.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:04.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.785+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:04.793 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:04.789+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14989 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:04.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:04.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:04.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:04.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:04.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980091fb0 con 0x7ff980050140 2021-03-09T10:14:04.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000ada0 con 0x7ff980050140 2021-03-09T10:14:04.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006300 secure :-1 s=READY pgs=4521 cs=0 l=1 rev1=1 rx=0x7ff9940142e0 tx=0x7ff994014190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:04.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020ec0 con 0x7ff980050140 2021-03-09T10:14:04.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405ee80 con 0x7ff980050140 2021-03-09T10:14:04.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.793+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:14:04.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.793+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:04.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.797+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:04.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:04.797+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:09.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980006300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:09.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980006300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:09.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006300 secure :-1 s=READY pgs=4521 cs=0 l=1 rev1=1 rx=0x7ff9940142e0 tx=0x7ff994014190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:09.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006300 secure :-1 s=READY pgs=4521 cs=0 l=1 rev1=1 rx=0x7ff9940142e0 tx=0x7ff994014190).stop 2021-03-09T10:14:09.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980006300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:09.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980006300 unknown :-1 s=CLOSED pgs=4521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:09.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:09.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:09.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:09.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005370 con 0x7ff99c0adc20 2021-03-09T10:14:09.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035f60 con 0x7ff99c0613a0 2021-03-09T10:14:09.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ada0 con 0x7ff980051540 2021-03-09T10:14:09.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:09.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:09.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:09.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:09.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:09.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:09.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:09.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:09.797 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:09.789+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15026 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:09.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:09.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800271a0 con 0x7ff980051540 2021-03-09T10:14:09.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091fb0 con 0x7ff980051540 2021-03-09T10:14:09.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4532 cs=0 l=1 rev1=1 rx=0x7ff994066770 tx=0x7ff9940aae20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:09.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.793+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980051540 2021-03-09T10:14:09.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.793+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994020220 con 0x7ff980051540 2021-03-09T10:14:09.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.797+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:14:09.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.797+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:09.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.797+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:09.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:09.797+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:14.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:14.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:14.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4532 cs=0 l=1 rev1=1 rx=0x7ff994066770 tx=0x7ff9940aae20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:14.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4532 cs=0 l=1 rev1=1 rx=0x7ff994066770 tx=0x7ff9940aae20).stop 2021-03-09T10:14:14.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:14.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=4532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:14.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:14.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:14.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:14.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037a70 con 0x7ff99c0adc20 2021-03-09T10:14:14.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800126e0 con 0x7ff99c0613a0 2021-03-09T10:14:14.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ada0 con 0x7ff980050140 2021-03-09T10:14:14.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:14.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:14.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:14.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.789+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:14.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.793+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:14.799 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:14.793+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15063 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:14.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:14.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.797+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:14.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:14.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.797+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:14.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800271a0 con 0x7ff980050140 2021-03-09T10:14:14.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091e20 con 0x7ff980050140 2021-03-09T10:14:14.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.797+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 secure :-1 s=READY pgs=4549 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c079860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:14.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.797+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980050140 2021-03-09T10:14:14.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.805+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980050140 2021-03-09T10:14:14.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.805+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a31a0 con 0x7ff980050140 2021-03-09T10:14:14.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.805+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:14.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.805+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:14.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:14.805+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:19.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:19.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:19.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 secure :-1 s=READY pgs=4549 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c079860).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:19.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 secure :-1 s=READY pgs=4549 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c079860).stop 2021-03-09T10:14:19.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:19.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 unknown :-1 s=CLOSED pgs=4549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:19.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:19.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:19.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:19.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:19.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980058830 con 0x7ff99c0adc20 2021-03-09T10:14:19.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:19.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:19.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037a70 con 0x7ff99c0613a0 2021-03-09T10:14:19.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:19.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800126e0 con 0x7ff980051540 2021-03-09T10:14:19.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.793+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:19.802 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:19.797+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15102 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:19.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:19.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.797+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:19.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:19.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.797+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:19.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027140 con 0x7ff980051540 2021-03-09T10:14:19.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091e20 con 0x7ff980051540 2021-03-09T10:14:19.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.797+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4553 cs=0 l=1 rev1=1 rx=0x7ff980094890 tx=0x7ff98c03c130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:19.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.801+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T10:14:19.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.801+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02c930 con 0x7ff980051540 2021-03-09T10:14:19.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.805+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00b330 con 0x7ff980051540 2021-03-09T10:14:19.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.805+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:19.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.805+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:19.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:19.805+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:24.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:24.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:24.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4553 cs=0 l=1 rev1=1 rx=0x7ff980094890 tx=0x7ff98c03c130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:24.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4553 cs=0 l=1 rev1=1 rx=0x7ff980094890 tx=0x7ff98c03c130).stop 2021-03-09T10:14:24.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:24.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=4553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:24.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:24.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:24.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:24.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022b70 con 0x7ff99c0adc20 2021-03-09T10:14:24.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057140 con 0x7ff99c0613a0 2021-03-09T10:14:24.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091e20 con 0x7ff980050140 2021-03-09T10:14:24.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:24.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:24.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:24.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:24.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:24.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:24.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:24.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:24.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:24.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035dc0 con 0x7ff980050140 2021-03-09T10:14:24.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.797+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091fb0 con 0x7ff980050140 2021-03-09T10:14:24.808 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:24.797+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15134 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:24.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.801+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 secure :-1 s=READY pgs=4565 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99400bf60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:24.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.801+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980050140 2021-03-09T10:14:24.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.801+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994012df0 con 0x7ff980050140 2021-03-09T10:14:24.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.809+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:14:24.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.809+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:24.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.809+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:24.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:24.809+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:29.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:14:29.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:29.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 secure :-1 s=READY pgs=4565 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99400bf60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:29.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 secure :-1 s=READY pgs=4565 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99400bf60).stop 2021-03-09T10:14:29.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:29.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 unknown :-1 s=CLOSED pgs=4565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:29.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:29.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:29.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:29.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091e70 con 0x7ff99c0adc20 2021-03-09T10:14:29.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0613a0 2021-03-09T10:14:29.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026f70 con 0x7ff980051540 2021-03-09T10:14:29.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:29.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:29.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:29.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:29.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.801+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:29.808 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:29.801+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15176 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:29.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.805+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:29.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.805+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:29.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.805+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:29.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.805+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:29.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.805+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980012450 con 0x7ff980051540 2021-03-09T10:14:29.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.805+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027140 con 0x7ff980051540 2021-03-09T10:14:29.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.805+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4579 cs=0 l=1 rev1=1 rx=0x7ff98c08dc90 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:29.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.809+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980051540 2021-03-09T10:14:29.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.813+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c046ef0 con 0x7ff980051540 2021-03-09T10:14:29.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.813+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079ba0 con 0x7ff980051540 2021-03-09T10:14:29.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.813+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:29.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.813+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:29.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:29.813+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:30.328 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:14:30.330 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:14:34.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.801+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:34.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.801+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:34.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.801+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4579 cs=0 l=1 rev1=1 rx=0x7ff98c08dc90 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:34.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.801+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4579 cs=0 l=1 rev1=1 rx=0x7ff98c08dc90 tx=0x7ff98c003e80).stop 2021-03-09T10:14:34.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.801+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:34.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.801+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=4579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:34.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.801+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:34.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:34.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:34.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091e70 con 0x7ff99c0adc20 2021-03-09T10:14:34.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013cb0 con 0x7ff99c0613a0 2021-03-09T10:14:34.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026f70 con 0x7ff980050140 2021-03-09T10:14:34.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:34.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:34.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:34.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:34.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:34.811 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:34.805+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15211 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:34.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:34.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:34.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:34.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:34.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035dc0 con 0x7ff980050140 2021-03-09T10:14:34.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980012450 con 0x7ff980050140 2021-03-09T10:14:34.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.805+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 secure :-1 s=READY pgs=4586 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c02cf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:34.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.809+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980050140 2021-03-09T10:14:34.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.809+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0171f0 con 0x7ff980050140 2021-03-09T10:14:34.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.813+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980050140 2021-03-09T10:14:34.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.813+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:34.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.813+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:34.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:34.813+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:39.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:14:39.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:39.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 secure :-1 s=READY pgs=4586 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c02cf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:39.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 secure :-1 s=READY pgs=4586 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c02cf40).stop 2021-03-09T10:14:39.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:39.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092140 unknown :-1 s=CLOSED pgs=4586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:39.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:39.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:39.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:39.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0adc20 2021-03-09T10:14:39.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026f00 con 0x7ff99c0613a0 2021-03-09T10:14:39.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027040 con 0x7ff980051540 2021-03-09T10:14:39.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:39.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:39.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:39.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.804+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:39.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.808+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:39.814 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:39.808+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15247 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:39.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.808+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:39.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.808+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:39.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.808+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:39.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.808+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:39.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.808+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800271b0 con 0x7ff980051540 2021-03-09T10:14:39.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.808+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980091fb0 con 0x7ff980051540 2021-03-09T10:14:39.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4603 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99401dbc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:39.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020ec0 con 0x7ff980051540 2021-03-09T10:14:39.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994030740 con 0x7ff980051540 2021-03-09T10:14:39.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:14:39.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.816+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:39.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.816+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:39.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:39.816+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:44.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:44.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:44.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4603 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99401dbc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:44.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4603 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99401dbc0).stop 2021-03-09T10:14:44.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:44.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=4603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:44.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:44.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:44.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:44.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0adc20 2021-03-09T10:14:44.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013bd0 con 0x7ff99c0613a0 2021-03-09T10:14:44.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013d10 con 0x7ff980050140 2021-03-09T10:14:44.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:44.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:44.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:44.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:44.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.808+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:44.817 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:44.808+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15275 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:44.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:44.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:44.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:44.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:44.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800924a0 con 0x7ff980050140 2021-03-09T10:14:44.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800271b0 con 0x7ff980050140 2021-03-09T10:14:44.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4612 cs=0 l=1 rev1=1 rx=0x7ff99403e440 tx=0x7ff99403e200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:44.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980050140 2021-03-09T10:14:44.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:14:44.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:14:44.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.820+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:44.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.820+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:44.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:44.820+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:49.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:49.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:49.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4612 cs=0 l=1 rev1=1 rx=0x7ff99403e440 tx=0x7ff99403e200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:49.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4612 cs=0 l=1 rev1=1 rx=0x7ff99403e440 tx=0x7ff99403e200).stop 2021-03-09T10:14:49.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:49.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=CLOSED pgs=4612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:49.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:49.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:49.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:49.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d260 con 0x7ff99c0adc20 2021-03-09T10:14:49.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800115d0 con 0x7ff99c0613a0 2021-03-09T10:14:49.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026ee0 con 0x7ff980051540 2021-03-09T10:14:49.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:49.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:49.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:49.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:49.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:49.820 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:49.812+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15321 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:49.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:49.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:49.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:49.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:49.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027140 con 0x7ff980051540 2021-03-09T10:14:49.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800924a0 con 0x7ff980051540 2021-03-09T10:14:49.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4617 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99400d2a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:49.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:14:49.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994031000 con 0x7ff980051540 2021-03-09T10:14:49.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:14:49.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.820+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:49.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.820+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:49.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:49.820+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:54.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:54.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:54.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4617 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99400d2a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:54.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 secure :-1 s=READY pgs=4617 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99400d2a0).stop 2021-03-09T10:14:54.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:54.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=4617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:54.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:54.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:54.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:54.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0adc20 2021-03-09T10:14:54.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800082e0 con 0x7ff99c0613a0 2021-03-09T10:14:54.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008420 con 0x7ff980050140 2021-03-09T10:14:54.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:54.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:54.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:54.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:54.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:54.822 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:54.816+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15351 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:54.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:54.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:54.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:54.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:54.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980008560 con 0x7ff980050140 2021-03-09T10:14:54.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027140 con 0x7ff980050140 2021-03-09T10:14:54.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.820+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4632 cs=0 l=1 rev1=1 rx=0x7ff994075500 tx=0x7ff9940666c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:54.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994031000 con 0x7ff980050140 2021-03-09T10:14:54.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994031000 con 0x7ff980050140 2021-03-09T10:14:54.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:14:54.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.824+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:54.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.828+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:54.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:54.828+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:14:59.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:59.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:59.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4632 cs=0 l=1 rev1=1 rx=0x7ff994075500 tx=0x7ff9940666c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:59.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4632 cs=0 l=1 rev1=1 rx=0x7ff994075500 tx=0x7ff9940666c0).stop 2021-03-09T10:14:59.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:59.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=CLOSED pgs=4632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:59.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:59.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:59.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:59.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800272e0 con 0x7ff99c0adc20 2021-03-09T10:14:59.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005500 con 0x7ff99c0613a0 2021-03-09T10:14:59.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026e20 con 0x7ff980051540 2021-03-09T10:14:59.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:14:59.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:59.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:14:59.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:59.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:59.827 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:14:59.820+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15394 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:14:59.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.820+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:59.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.820+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:59.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.820+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:59.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.820+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:59.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.820+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800924a0 con 0x7ff980051540 2021-03-09T10:14:59.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.820+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980008560 con 0x7ff980051540 2021-03-09T10:14:59.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.820+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 secure :-1 s=READY pgs=4639 cs=0 l=1 rev1=1 rx=0x7ff994093180 tx=0x7ff994064ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:59.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff980051540 2021-03-09T10:14:59.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d5e0 con 0x7ff980051540 2021-03-09T10:14:59.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:14:59.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.824+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:59.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.828+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:59.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:14:59.828+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:00.367 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:15:00.369 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:15:04.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:04.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:04.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 secure :-1 s=READY pgs=4639 cs=0 l=1 rev1=1 rx=0x7ff994093180 tx=0x7ff994064ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:04.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 secure :-1 s=READY pgs=4639 cs=0 l=1 rev1=1 rx=0x7ff994093180 tx=0x7ff994064ef0).stop 2021-03-09T10:15:04.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:04.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980005010 unknown :-1 s=CLOSED pgs=4639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:04.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:04.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:04.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:04.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800272e0 con 0x7ff99c0adc20 2021-03-09T10:15:04.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0613a0 2021-03-09T10:15:04.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092630 con 0x7ff980050140 2021-03-09T10:15:04.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:04.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:04.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:04.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:04.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:04.828 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:04.820+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15431 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:04.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:04.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:04.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:04.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:04.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800927a0 con 0x7ff980050140 2021-03-09T10:15:04.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.820+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980012450 con 0x7ff980050140 2021-03-09T10:15:04.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.824+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4649 cs=0 l=1 rev1=1 rx=0x7ff98c00a2e0 tx=0x7ff98c08def0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:04.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.828+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:15:04.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.832+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08dc70 con 0x7ff980050140 2021-03-09T10:15:04.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.832+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980050140 2021-03-09T10:15:04.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.832+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:04.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.836+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:04.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:04.836+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:09.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:09.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:09.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4649 cs=0 l=1 rev1=1 rx=0x7ff98c00a2e0 tx=0x7ff98c08def0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:09.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4649 cs=0 l=1 rev1=1 rx=0x7ff98c00a2e0 tx=0x7ff98c08def0).stop 2021-03-09T10:15:09.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:09.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=CLOSED pgs=4649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:09.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:09.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:09.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:09.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:09.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d840 con 0x7ff99c0adc20 2021-03-09T10:15:09.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d410 con 0x7ff99c0613a0 2021-03-09T10:15:09.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:09.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092630 con 0x7ff980051540 2021-03-09T10:15:09.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:09.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:09.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:09.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:09.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:09.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:09.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:09.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800272e0 con 0x7ff980051540 2021-03-09T10:15:09.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800927a0 con 0x7ff980051540 2021-03-09T10:15:09.833 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:09.824+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15469 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:09.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.824+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092930 secure :-1 s=READY pgs=4660 cs=0 l=1 rev1=1 rx=0x7ff98c0a9930 tx=0x7ff98c0a9850).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:09.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.828+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980051540 2021-03-09T10:15:09.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.828+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980051540 2021-03-09T10:15:09.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.832+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bcd0 con 0x7ff980051540 2021-03-09T10:15:09.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.832+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:09.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.832+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:09.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:09.832+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:14.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:14.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:14.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092930 secure :-1 s=READY pgs=4660 cs=0 l=1 rev1=1 rx=0x7ff98c0a9930 tx=0x7ff98c0a9850).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:14.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092930 secure :-1 s=READY pgs=4660 cs=0 l=1 rev1=1 rx=0x7ff98c0a9930 tx=0x7ff98c0a9850).stop 2021-03-09T10:15:14.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:14.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092930 unknown :-1 s=CLOSED pgs=4660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:14.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:14.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:14.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:14.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d410 con 0x7ff99c0adc20 2021-03-09T10:15:14.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d840 con 0x7ff99c0613a0 2021-03-09T10:15:14.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800927d0 con 0x7ff980050140 2021-03-09T10:15:14.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:14.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:14.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:14.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:14.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.824+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:14.834 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:14.828+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15508 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:14.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.828+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:14.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.828+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:14.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.828+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:14.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.828+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:14.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.828+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035dc0 con 0x7ff980050140 2021-03-09T10:15:14.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.828+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800924a0 con 0x7ff980050140 2021-03-09T10:15:14.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.832+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4675 cs=0 l=1 rev1=1 rx=0x7ff98008b420 tx=0x7ff994066650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:14.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.832+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff980050140 2021-03-09T10:15:14.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.836+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff980050140 2021-03-09T10:15:14.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.836+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:15:14.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.836+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:14.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.840+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:14.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:14.840+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:19.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:19.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:19.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4675 cs=0 l=1 rev1=1 rx=0x7ff98008b420 tx=0x7ff994066650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:19.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4675 cs=0 l=1 rev1=1 rx=0x7ff98008b420 tx=0x7ff994066650).stop 2021-03-09T10:15:19.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:19.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=CLOSED pgs=4675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:19.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:19.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:19.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:19.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:19.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:19.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037fb0 con 0x7ff99c0adc20 2021-03-09T10:15:19.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b760 con 0x7ff99c0613a0 2021-03-09T10:15:19.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d3c0 con 0x7ff980051540 2021-03-09T10:15:19.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:19.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.828+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:19.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.832+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:19.838 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:19.832+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15529 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:19.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.832+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:19.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.832+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:19.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.832+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:19.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.832+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:19.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.832+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980024450 con 0x7ff980051540 2021-03-09T10:15:19.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.832+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800924a0 con 0x7ff980051540 2021-03-09T10:15:19.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.832+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 secure :-1 s=READY pgs=4684 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940813b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:19.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:15:19.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:15:19.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:15:19.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.840+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:19.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.840+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:19.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:19.840+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:24.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:24.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:24.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 secure :-1 s=READY pgs=4684 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940813b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:24.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 secure :-1 s=READY pgs=4684 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940813b0).stop 2021-03-09T10:15:24.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:24.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 unknown :-1 s=CLOSED pgs=4684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:24.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:24.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:24.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:24.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d3c0 con 0x7ff99c0adc20 2021-03-09T10:15:24.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003b20 con 0x7ff99c0613a0 2021-03-09T10:15:24.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800924a0 con 0x7ff980050140 2021-03-09T10:15:24.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:24.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:24.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:24.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:24.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.832+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:24.841 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:24.832+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15580 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:24.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.836+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:24.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.836+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:24.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.836+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:24.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.836+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:24.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.836+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035dc0 con 0x7ff980050140 2021-03-09T10:15:24.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.836+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980024450 con 0x7ff980050140 2021-03-09T10:15:24.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.836+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4695 cs=0 l=1 rev1=1 rx=0x7ff980058310 tx=0x7ff99406fef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:24.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069f70 con 0x7ff980050140 2021-03-09T10:15:24.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.844+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069f70 con 0x7ff980050140 2021-03-09T10:15:24.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.844+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:15:24.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.844+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:24.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.848+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:24.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:24.848+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:29.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:29.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:29.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4695 cs=0 l=1 rev1=1 rx=0x7ff980058310 tx=0x7ff99406fef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:29.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 secure :-1 s=READY pgs=4695 cs=0 l=1 rev1=1 rx=0x7ff980058310 tx=0x7ff99406fef0).stop 2021-03-09T10:15:29.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:29.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091fb0 unknown :-1 s=CLOSED pgs=4695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:29.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800924a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:29.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:29.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:29.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d3c0 con 0x7ff99c0adc20 2021-03-09T10:15:29.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800304f0 con 0x7ff99c0613a0 2021-03-09T10:15:29.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003b20 con 0x7ff980051540 2021-03-09T10:15:29.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:29.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:29.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:29.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:29.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.836+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800924a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:29.843 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:29.836+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15616 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:29.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:29.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:29.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:29.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:29.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035dc0 con 0x7ff980051540 2021-03-09T10:15:29.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980024450 con 0x7ff980051540 2021-03-09T10:15:29.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800924a0 secure :-1 s=READY pgs=4710 cs=0 l=1 rev1=1 rx=0x7ff98000b4d0 tx=0x7ff994097080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:29.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.844+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:15:29.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:15:29.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:15:29.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.848+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:29.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.848+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:29.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:29.848+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:30.412 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:15:30.415 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:15:34.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800924a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:34.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800924a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:34.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800924a0 secure :-1 s=READY pgs=4710 cs=0 l=1 rev1=1 rx=0x7ff98000b4d0 tx=0x7ff994097080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:34.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800924a0 secure :-1 s=READY pgs=4710 cs=0 l=1 rev1=1 rx=0x7ff98000b4d0 tx=0x7ff994097080).stop 2021-03-09T10:15:34.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800924a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:34.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800924a0 unknown :-1 s=CLOSED pgs=4710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:34.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:34.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:34.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:34.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d3c0 con 0x7ff99c0adc20 2021-03-09T10:15:34.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003b20 con 0x7ff99c0613a0 2021-03-09T10:15:34.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043570 con 0x7ff980050140 2021-03-09T10:15:34.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:34.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:34.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:34.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:34.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:34.847 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:34.840+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15655 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:34.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:34.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:34.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:34.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:34.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800123c0 con 0x7ff980050140 2021-03-09T10:15:34.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980024450 con 0x7ff980050140 2021-03-09T10:15:34.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ed0 secure :-1 s=READY pgs=4713 cs=0 l=1 rev1=1 rx=0x7ff994097470 tx=0x7ff994068250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:34.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.844+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b13f0 con 0x7ff980050140 2021-03-09T10:15:34.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.844+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940b11c0 con 0x7ff980050140 2021-03-09T10:15:34.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:15:34.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.848+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:34.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.848+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:34.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:34.848+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:39.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:39.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:39.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ed0 secure :-1 s=READY pgs=4713 cs=0 l=1 rev1=1 rx=0x7ff994097470 tx=0x7ff994068250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:39.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ed0 secure :-1 s=READY pgs=4713 cs=0 l=1 rev1=1 rx=0x7ff994097470 tx=0x7ff994068250).stop 2021-03-09T10:15:39.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:39.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ed0 unknown :-1 s=CLOSED pgs=4713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:39.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:39.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800267a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:39.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:39.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003b20 con 0x7ff99c0adc20 2021-03-09T10:15:39.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d3a0 con 0x7ff99c0613a0 2021-03-09T10:15:39.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026c90 con 0x7ff980051540 2021-03-09T10:15:39.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:39.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:39.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:39.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.840+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:39.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.844+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800267a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:39.850 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:39.844+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15688 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:39.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.844+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:39.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.844+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:39.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.844+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:39.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.844+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:39.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.844+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800924a0 con 0x7ff980051540 2021-03-09T10:15:39.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.844+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800272e0 con 0x7ff980051540 2021-03-09T10:15:39.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.844+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800267a0 secure :-1 s=READY pgs=4727 cs=0 l=1 rev1=1 rx=0x7ff980094e10 tx=0x7ff98c00bfb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:39.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980051540 2021-03-09T10:15:39.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980051540 2021-03-09T10:15:39.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.852+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:15:39.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.852+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:39.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.852+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:39.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:39.852+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:44.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800267a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:44.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800267a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:44.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800267a0 secure :-1 s=READY pgs=4727 cs=0 l=1 rev1=1 rx=0x7ff980094e10 tx=0x7ff98c00bfb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:44.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800267a0 secure :-1 s=READY pgs=4727 cs=0 l=1 rev1=1 rx=0x7ff980094e10 tx=0x7ff98c00bfb0).stop 2021-03-09T10:15:44.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800267a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:44.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800267a0 unknown :-1 s=CLOSED pgs=4727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:44.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:44.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:44.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:44.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800272e0 con 0x7ff99c0adc20 2021-03-09T10:15:44.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003b20 con 0x7ff99c0613a0 2021-03-09T10:15:44.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff980050140 2021-03-09T10:15:44.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:44.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:44.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:44.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:44.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.844+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:44.853 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:44.848+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15729 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:44.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.848+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:44.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.848+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:44.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.848+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:44.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.848+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:44.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.848+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800121d0 con 0x7ff980050140 2021-03-09T10:15:44.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.848+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800924a0 con 0x7ff980050140 2021-03-09T10:15:44.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.848+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ce0 secure :-1 s=READY pgs=4740 cs=0 l=1 rev1=1 rx=0x7ff98c00a2e0 tx=0x7ff98c017210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:44.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:15:44.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980050140 2021-03-09T10:15:44.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.856+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980050140 2021-03-09T10:15:44.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.856+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:44.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.856+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:44.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:44.856+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:49.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:49.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:49.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ce0 secure :-1 s=READY pgs=4740 cs=0 l=1 rev1=1 rx=0x7ff98c00a2e0 tx=0x7ff98c017210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:49.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ce0 secure :-1 s=READY pgs=4740 cs=0 l=1 rev1=1 rx=0x7ff98c00a2e0 tx=0x7ff98c017210).stop 2021-03-09T10:15:49.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:49.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011ce0 unknown :-1 s=CLOSED pgs=4740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:49.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:49.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:49.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:49.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0adc20 2021-03-09T10:15:49.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027140 con 0x7ff99c0613a0 2021-03-09T10:15:49.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027280 con 0x7ff980051540 2021-03-09T10:15:49.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:49.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:49.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:49.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:49.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.848+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:49.857 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:49.848+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15765 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:49.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:49.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.852+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:49.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:49.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.852+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:49.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800924a0 con 0x7ff980051540 2021-03-09T10:15:49.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800123c0 con 0x7ff980051540 2021-03-09T10:15:49.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.852+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026700 secure :-1 s=READY pgs=4747 cs=0 l=1 rev1=1 rx=0x7ff9940a90b0 tx=0x7ff994081380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:49.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.852+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994075340 con 0x7ff980051540 2021-03-09T10:15:49.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.852+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994075110 con 0x7ff980051540 2021-03-09T10:15:49.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.856+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:15:49.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.856+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:49.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.860+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:49.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:49.860+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:54.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:54.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:54.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026700 secure :-1 s=READY pgs=4747 cs=0 l=1 rev1=1 rx=0x7ff9940a90b0 tx=0x7ff994081380).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:54.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026700 secure :-1 s=READY pgs=4747 cs=0 l=1 rev1=1 rx=0x7ff9940a90b0 tx=0x7ff994081380).stop 2021-03-09T10:15:54.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:54.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026700 unknown :-1 s=CLOSED pgs=4747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:54.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:54.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:54.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:54.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0adc20 2021-03-09T10:15:54.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f160 con 0x7ff99c0613a0 2021-03-09T10:15:54.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f2a0 con 0x7ff980050140 2021-03-09T10:15:54.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:54.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:54.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:54.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:54.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:54.860 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:54.852+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15803 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:54.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:54.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:54.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:54.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:54.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027140 con 0x7ff980050140 2021-03-09T10:15:54.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800924a0 con 0x7ff980050140 2021-03-09T10:15:54.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.852+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4756 cs=0 l=1 rev1=1 rx=0x7ff994036010 tx=0x7ff994093140).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:54.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.856+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994075340 con 0x7ff980050140 2021-03-09T10:15:54.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.860+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994075340 con 0x7ff980050140 2021-03-09T10:15:54.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.860+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:15:54.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.860+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:54.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.864+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:54.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:54.864+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:15:59.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:59.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:59.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4756 cs=0 l=1 rev1=1 rx=0x7ff994036010 tx=0x7ff994093140).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:59.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4756 cs=0 l=1 rev1=1 rx=0x7ff994036010 tx=0x7ff994093140).stop 2021-03-09T10:15:59.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:59.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=CLOSED pgs=4756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:59.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:59.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:59.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:59.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800266d0 con 0x7ff99c0adc20 2021-03-09T10:15:59.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026810 con 0x7ff99c0613a0 2021-03-09T10:15:59.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026950 con 0x7ff980051540 2021-03-09T10:15:59.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:15:59.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:15:59.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:59.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:59.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:59.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:59.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:59.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:59.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:59.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026a90 con 0x7ff980051540 2021-03-09T10:15:59.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027140 con 0x7ff980051540 2021-03-09T10:15:59.865 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:15:59.855+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15836 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:15:59.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 secure :-1 s=READY pgs=4767 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99400bf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:59.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994064a10 con 0x7ff980051540 2021-03-09T10:15:59.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994064830 con 0x7ff980051540 2021-03-09T10:15:59.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:15:59.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.863+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:59.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.863+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:59.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:15:59.863+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:00.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:16:00.571 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:16:04.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:04.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:04.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 secure :-1 s=READY pgs=4767 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99400bf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:04.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 secure :-1 s=READY pgs=4767 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99400bf40).stop 2021-03-09T10:16:04.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c7f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:04.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 unknown :-1 s=CLOSED pgs=4767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:04.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:04.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:04.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:04.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004e500 con 0x7ff99c0adc20 2021-03-09T10:16:04.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b080 con 0x7ff99c0613a0 2021-03-09T10:16:04.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094350 con 0x7ff980050140 2021-03-09T10:16:04.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:04.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:04.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:04.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:04.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:04.866 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:04.859+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15877 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:04.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:04.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:04.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:04.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:04.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800272e0 con 0x7ff980050140 2021-03-09T10:16:04.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026a90 con 0x7ff980050140 2021-03-09T10:16:04.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4777 cs=0 l=1 rev1=1 rx=0x7ff994039040 tx=0x7ff994064cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:04.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994071810 con 0x7ff980050140 2021-03-09T10:16:04.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994064830 con 0x7ff980050140 2021-03-09T10:16:04.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980050140 2021-03-09T10:16:04.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.871+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:04.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.871+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:04.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:04.871+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:09.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:16:09.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:09.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4777 cs=0 l=1 rev1=1 rx=0x7ff994039040 tx=0x7ff994064cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:09.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4777 cs=0 l=1 rev1=1 rx=0x7ff994039040 tx=0x7ff994064cf0).stop 2021-03-09T10:16:09.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:09.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=CLOSED pgs=4777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:09.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:09.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:09.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:09.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026680 con 0x7ff99c0adc20 2021-03-09T10:16:09.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800267c0 con 0x7ff99c0613a0 2021-03-09T10:16:09.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a720 con 0x7ff980051540 2021-03-09T10:16:09.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:09.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:09.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:09.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:09.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.859+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:09.869 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:09.863+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15913 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:09.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.863+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:09.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.863+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:09.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.863+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:09.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.863+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:09.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.863+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000a860 con 0x7ff980051540 2021-03-09T10:16:09.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.863+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800121d0 con 0x7ff980051540 2021-03-09T10:16:09.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.863+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 secure :-1 s=READY pgs=4790 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c0066c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:09.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T10:16:09.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c09f170 con 0x7ff980051540 2021-03-09T10:16:09.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c09f170 con 0x7ff980051540 2021-03-09T10:16:09.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.871+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:09.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.871+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:09.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:09.871+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:14.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:14.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:14.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 secure :-1 s=READY pgs=4790 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c0066c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:14.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 secure :-1 s=READY pgs=4790 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c0066c0).stop 2021-03-09T10:16:14.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c7f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:14.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c7f0 unknown :-1 s=CLOSED pgs=4790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:14.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:14.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:14.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:14.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b080 con 0x7ff99c0adc20 2021-03-09T10:16:14.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d6b0 con 0x7ff99c0613a0 2021-03-09T10:16:14.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012370 con 0x7ff980050140 2021-03-09T10:16:14.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:14.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:14.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:14.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:14.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:14.873 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:14.863+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15951 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:14.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:14.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:14.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:14.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:14.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035dc0 con 0x7ff980050140 2021-03-09T10:16:14.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800272e0 con 0x7ff980050140 2021-03-09T10:16:14.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4802 cs=0 l=1 rev1=1 rx=0x7ff980050df0 tx=0x7ff994071d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:14.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095ce0 con 0x7ff980050140 2021-03-09T10:16:14.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994071810 con 0x7ff980050140 2021-03-09T10:16:14.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:16:14.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.875+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:14.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.875+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:14.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:14.875+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:19.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:19.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:19.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4802 cs=0 l=1 rev1=1 rx=0x7ff980050df0 tx=0x7ff994071d10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:19.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4802 cs=0 l=1 rev1=1 rx=0x7ff980050df0 tx=0x7ff994071d10).stop 2021-03-09T10:16:19.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:19.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=CLOSED pgs=4802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:19.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:19.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:19.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:19.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce60 con 0x7ff99c0adc20 2021-03-09T10:16:19.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cb10 con 0x7ff99c0613a0 2021-03-09T10:16:19.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff980051540 2021-03-09T10:16:19.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:19.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:19.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:19.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:19.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:19.876 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:19.867+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15987 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:19.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:19.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:19.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:19.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:19.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026650 con 0x7ff980051540 2021-03-09T10:16:19.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800272e0 con 0x7ff980051540 2021-03-09T10:16:19.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c620 secure :-1 s=READY pgs=4809 cs=0 l=1 rev1=1 rx=0x7ff99400d530 tx=0x7ff99403e1b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:19.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069020 con 0x7ff980051540 2021-03-09T10:16:19.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095ce0 con 0x7ff980051540 2021-03-09T10:16:19.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:16:19.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.875+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:19.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.879+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:19.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:19.879+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:24.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:24.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:24.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c620 secure :-1 s=READY pgs=4809 cs=0 l=1 rev1=1 rx=0x7ff99400d530 tx=0x7ff99403e1b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:24.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c620 secure :-1 s=READY pgs=4809 cs=0 l=1 rev1=1 rx=0x7ff99400d530 tx=0x7ff99403e1b0).stop 2021-03-09T10:16:24.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:24.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c620 unknown :-1 s=CLOSED pgs=4809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:24.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:24.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:24.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:24.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:24.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:24.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035da0 con 0x7ff99c0adc20 2021-03-09T10:16:24.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800121b0 con 0x7ff99c0613a0 2021-03-09T10:16:24.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800122f0 con 0x7ff980050140 2021-03-09T10:16:24.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:24.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:24.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:24.879 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:24.871+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16025 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:24.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:24.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:24.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:24.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:24.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092460 con 0x7ff980050140 2021-03-09T10:16:24.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.871+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000a860 con 0x7ff980050140 2021-03-09T10:16:24.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4823 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c07da90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:24.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075030 con 0x7ff980050140 2021-03-09T10:16:24.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980050140 2021-03-09T10:16:24.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980050140 2021-03-09T10:16:24.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.883+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:24.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.883+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:24.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:24.883+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:29.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:29.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:29.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4823 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c07da90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:29.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 secure :-1 s=READY pgs=4823 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c07da90).stop 2021-03-09T10:16:29.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980011cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:29.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980011cc0 unknown :-1 s=CLOSED pgs=4823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:29.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:29.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:29.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:29.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cb40 con 0x7ff99c0adc20 2021-03-09T10:16:29.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cc80 con 0x7ff99c0613a0 2021-03-09T10:16:29.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800925f0 con 0x7ff980051540 2021-03-09T10:16:29.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:29.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:29.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:29.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:29.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:29.883 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:29.875+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16061 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:29.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:29.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:29.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:29.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.875+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:29.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.879+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092730 con 0x7ff980051540 2021-03-09T10:16:29.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.879+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026650 con 0x7ff980051540 2021-03-09T10:16:29.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.879+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c5c0 secure :-1 s=READY pgs=4836 cs=0 l=1 rev1=1 rx=0x7ff994075620 tx=0x7ff99403e140).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:29.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.879+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980051540 2021-03-09T10:16:29.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994071a60 con 0x7ff980051540 2021-03-09T10:16:29.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:16:29.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.883+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:29.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.887+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:29.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:29.887+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:30.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:16:30.607 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:16:34.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:34.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:34.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c5c0 secure :-1 s=READY pgs=4836 cs=0 l=1 rev1=1 rx=0x7ff994075620 tx=0x7ff99403e140).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:34.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c5c0 secure :-1 s=READY pgs=4836 cs=0 l=1 rev1=1 rx=0x7ff994075620 tx=0x7ff99403e140).stop 2021-03-09T10:16:34.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:34.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c5c0 unknown :-1 s=CLOSED pgs=4836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:34.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:34.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:34.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:34.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800925f0 con 0x7ff99c0adc20 2021-03-09T10:16:34.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088a70 con 0x7ff99c0613a0 2021-03-09T10:16:34.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035da0 con 0x7ff980050140 2021-03-09T10:16:34.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:34.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:34.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:34.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:34.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.875+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:34.886 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:34.879+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16098 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:34.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:34.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.879+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:34.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:34.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.879+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:34.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980012330 con 0x7ff980050140 2021-03-09T10:16:34.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092460 con 0x7ff980050140 2021-03-09T10:16:34.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.879+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 secure :-1 s=READY pgs=4841 cs=0 l=1 rev1=1 rx=0x7ff98c00c490 tx=0x7ff98c07da20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:34.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.879+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:16:34.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.879+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04c140 con 0x7ff980050140 2021-03-09T10:16:34.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980050140 2021-03-09T10:16:34.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.883+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:34.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.887+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:34.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:34.887+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:39.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:39.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:39.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 secure :-1 s=READY pgs=4841 cs=0 l=1 rev1=1 rx=0x7ff98c00c490 tx=0x7ff98c07da20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:39.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 secure :-1 s=READY pgs=4841 cs=0 l=1 rev1=1 rx=0x7ff98c00c490 tx=0x7ff98c07da20).stop 2021-03-09T10:16:39.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800928c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:39.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 unknown :-1 s=CLOSED pgs=4841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:39.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:39.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:39.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:39.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800037b0 con 0x7ff99c0adc20 2021-03-09T10:16:39.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088a70 con 0x7ff99c0613a0 2021-03-09T10:16:39.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035da0 con 0x7ff980051540 2021-03-09T10:16:39.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:39.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:39.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:39.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:39.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.879+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:39.889 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:39.879+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16135 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:39.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:39.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:39.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:39.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:39.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000af80 con 0x7ff980051540 2021-03-09T10:16:39.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092730 con 0x7ff980051540 2021-03-09T10:16:39.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c520 secure :-1 s=READY pgs=4856 cs=0 l=1 rev1=1 rx=0x7ff994066730 tx=0x7ff994014250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:39.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980051540 2021-03-09T10:16:39.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994074810 con 0x7ff980051540 2021-03-09T10:16:39.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:16:39.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.891+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:39.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.895+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:39.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:39.895+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:44.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:44.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:44.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c520 secure :-1 s=READY pgs=4856 cs=0 l=1 rev1=1 rx=0x7ff994066730 tx=0x7ff994014250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:44.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c520 secure :-1 s=READY pgs=4856 cs=0 l=1 rev1=1 rx=0x7ff994066730 tx=0x7ff994014250).stop 2021-03-09T10:16:44.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:44.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c520 unknown :-1 s=CLOSED pgs=4856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:44.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:44.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98001a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:44.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:44.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088a70 con 0x7ff99c0adc20 2021-03-09T10:16:44.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800037b0 con 0x7ff99c0613a0 2021-03-09T10:16:44.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035da0 con 0x7ff980050140 2021-03-09T10:16:44.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:44.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:44.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:44.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:44.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.883+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98001a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:44.892 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:44.883+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16172 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:44.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.887+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:44.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.887+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:44.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.887+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:44.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.887+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:44.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.887+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092730 con 0x7ff980050140 2021-03-09T10:16:44.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.887+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980012330 con 0x7ff980050140 2021-03-09T10:16:44.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.891+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98001a1c0 secure :-1 s=READY pgs=4867 cs=0 l=1 rev1=1 rx=0x7ff98000b7f0 tx=0x7ff98c07e150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:44.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:16:44.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff980050140 2021-03-09T10:16:44.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980050140 2021-03-09T10:16:44.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.895+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:44.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.899+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:44.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:44.899+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:49.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98001a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:49.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98001a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:49.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98001a1c0 secure :-1 s=READY pgs=4867 cs=0 l=1 rev1=1 rx=0x7ff98000b7f0 tx=0x7ff98c07e150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:49.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98001a1c0 secure :-1 s=READY pgs=4867 cs=0 l=1 rev1=1 rx=0x7ff98000b7f0 tx=0x7ff98c07e150).stop 2021-03-09T10:16:49.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98001a1c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:49.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98001a1c0 unknown :-1 s=CLOSED pgs=4867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:49.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:49.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:49.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:49.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094b70 con 0x7ff99c0adc20 2021-03-09T10:16:49.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0613a0 2021-03-09T10:16:49.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800037b0 con 0x7ff980051540 2021-03-09T10:16:49.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:49.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:49.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:49.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:49.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:49.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:49.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:49.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:49.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:49.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026650 con 0x7ff980051540 2021-03-09T10:16:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000af80 con 0x7ff980051540 2021-03-09T10:16:49.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c4a0 secure :-1 s=READY pgs=4873 cs=0 l=1 rev1=1 rx=0x7ff994095630 tx=0x7ff9940813b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:49.897 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:49.887+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16209 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:16:49.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:16:49.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:16:49.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.895+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:49.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.895+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:49.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:49.895+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:54.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:16:54.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:54.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c4a0 secure :-1 s=READY pgs=4873 cs=0 l=1 rev1=1 rx=0x7ff994095630 tx=0x7ff9940813b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:54.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c4a0 secure :-1 s=READY pgs=4873 cs=0 l=1 rev1=1 rx=0x7ff994095630 tx=0x7ff9940813b0).stop 2021-03-09T10:16:54.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:54.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c4a0 unknown :-1 s=CLOSED pgs=4873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:54.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:54.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:54.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:54.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0adc20 2021-03-09T10:16:54.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026650 con 0x7ff99c0613a0 2021-03-09T10:16:54.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a360 con 0x7ff980050140 2021-03-09T10:16:54.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:54.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:54.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:54.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:54.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:54.898 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:54.891+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16247 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:54.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:54.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:54.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:54.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:54.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001a4a0 con 0x7ff980050140 2021-03-09T10:16:54.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.891+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000af80 con 0x7ff980050140 2021-03-09T10:16:54.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 secure :-1 s=READY pgs=4889 cs=0 l=1 rev1=1 rx=0x7ff99400ba70 tx=0x7ff994027120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:54.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940af8a0 con 0x7ff980050140 2021-03-09T10:16:54.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940af000 con 0x7ff980050140 2021-03-09T10:16:54.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.899+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af8a0 con 0x7ff980050140 2021-03-09T10:16:54.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.899+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:54.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.899+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:54.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:54.899+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:16:59.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:59.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:59.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 secure :-1 s=READY pgs=4889 cs=0 l=1 rev1=1 rx=0x7ff99400ba70 tx=0x7ff994027120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:59.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 secure :-1 s=READY pgs=4889 cs=0 l=1 rev1=1 rx=0x7ff99400ba70 tx=0x7ff994027120).stop 2021-03-09T10:16:59.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800928c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:59.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 unknown :-1 s=CLOSED pgs=4889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:59.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:59.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:59.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:59.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a360 con 0x7ff99c0adc20 2021-03-09T10:16:59.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026650 con 0x7ff99c0613a0 2021-03-09T10:16:59.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c7a0 con 0x7ff980051540 2021-03-09T10:16:59.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:16:59.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:16:59.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:59.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:59.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:59.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:59.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:59.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:59.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:59.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800570f0 con 0x7ff980051540 2021-03-09T10:16:59.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001a4a0 con 0x7ff980051540 2021-03-09T10:16:59.904 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:16:59.895+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16283 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:16:59.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c2b0 secure :-1 s=READY pgs=4895 cs=0 l=1 rev1=1 rx=0x7ff994098e90 tx=0x7ff994098d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:59.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.899+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994006be0 con 0x7ff980051540 2021-03-09T10:16:59.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.899+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980051540 2021-03-09T10:16:59.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.903+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af8a0 con 0x7ff980051540 2021-03-09T10:16:59.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.903+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:59.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.903+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:59.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:16:59.903+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:00.652 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:17:00.655 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:17:04.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:04.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:04.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c2b0 secure :-1 s=READY pgs=4895 cs=0 l=1 rev1=1 rx=0x7ff994098e90 tx=0x7ff994098d40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:04.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c2b0 secure :-1 s=READY pgs=4895 cs=0 l=1 rev1=1 rx=0x7ff994098e90 tx=0x7ff994098d40).stop 2021-03-09T10:17:04.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008c2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:04.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008c2b0 unknown :-1 s=CLOSED pgs=4895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:04.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:04.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:04.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:04.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980033760 con 0x7ff99c0adc20 2021-03-09T10:17:04.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010dc0 con 0x7ff99c0613a0 2021-03-09T10:17:04.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800359e0 con 0x7ff980050140 2021-03-09T10:17:04.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:04.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:04.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:04.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.895+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:04.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.899+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:04.905 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:04.899+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16321 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:04.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.899+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:04.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.899+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:04.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.899+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:04.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.899+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:04.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.899+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035b50 con 0x7ff980050140 2021-03-09T10:17:04.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.899+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092730 con 0x7ff980050140 2021-03-09T10:17:04.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.899+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 secure :-1 s=READY pgs=4910 cs=0 l=1 rev1=1 rx=0x7ff98001ba00 tx=0x7ff98c0a2ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:04.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.903+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980050140 2021-03-09T10:17:04.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.903+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980050140 2021-03-09T10:17:04.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.907+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980050140 2021-03-09T10:17:04.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.907+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:04.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.907+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:04.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:04.907+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:09.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:09.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:09.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 secure :-1 s=READY pgs=4910 cs=0 l=1 rev1=1 rx=0x7ff98001ba00 tx=0x7ff98c0a2ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:09.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 secure :-1 s=READY pgs=4910 cs=0 l=1 rev1=1 rx=0x7ff98001ba00 tx=0x7ff98c0a2ef0).stop 2021-03-09T10:17:09.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800928c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:09.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800928c0 unknown :-1 s=CLOSED pgs=4910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:09.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:09.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:09.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:09.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b0e0 con 0x7ff99c0adc20 2021-03-09T10:17:09.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004bed0 con 0x7ff99c0613a0 2021-03-09T10:17:09.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c820 con 0x7ff980051540 2021-03-09T10:17:09.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:09.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:09.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:09.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:09.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.899+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:09.911 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:09.899+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16357 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:09.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.903+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:09.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.903+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:09.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.903+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:09.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.903+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:09.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.903+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092730 con 0x7ff980051540 2021-03-09T10:17:09.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.903+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800570f0 con 0x7ff980051540 2021-03-09T10:17:09.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.903+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 secure :-1 s=READY pgs=4917 cs=0 l=1 rev1=1 rx=0x7ff994027040 tx=0x7ff9940114c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:09.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.903+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069030 con 0x7ff980051540 2021-03-09T10:17:09.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.903+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980051540 2021-03-09T10:17:09.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.907+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:17:09.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.907+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:09.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.907+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:09.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:09.907+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:14.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:14.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:14.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 secure :-1 s=READY pgs=4917 cs=0 l=1 rev1=1 rx=0x7ff994027040 tx=0x7ff9940114c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:14.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 secure :-1 s=READY pgs=4917 cs=0 l=1 rev1=1 rx=0x7ff994027040 tx=0x7ff9940114c0).stop 2021-03-09T10:17:14.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:14.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 unknown :-1 s=CLOSED pgs=4917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:14.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:14.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:14.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:14.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0adc20 2021-03-09T10:17:14.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800536f0 con 0x7ff99c0613a0 2021-03-09T10:17:14.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094700 con 0x7ff980050140 2021-03-09T10:17:14.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:14.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:14.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:14.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:14.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:14.911 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:14.902+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16395 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:14.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:14.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:14.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:14.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:14.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980033760 con 0x7ff980050140 2021-03-09T10:17:14.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035b50 con 0x7ff980050140 2021-03-09T10:17:14.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.906+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056f50 secure :-1 s=READY pgs=4928 cs=0 l=1 rev1=1 rx=0x7ff98c04b8f0 tx=0x7ff98c02cf00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:14.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:17:14.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c052130 con 0x7ff980050140 2021-03-09T10:17:14.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980050140 2021-03-09T10:17:14.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.910+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:14.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.914+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:14.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:14.914+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:19.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:19.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:19.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056f50 secure :-1 s=READY pgs=4928 cs=0 l=1 rev1=1 rx=0x7ff98c04b8f0 tx=0x7ff98c02cf00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:19.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056f50 secure :-1 s=READY pgs=4928 cs=0 l=1 rev1=1 rx=0x7ff98c04b8f0 tx=0x7ff98c02cf00).stop 2021-03-09T10:17:19.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:19.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056f50 unknown :-1 s=CLOSED pgs=4928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:19.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:19.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:19.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:19.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0adc20 2021-03-09T10:17:19.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f430 con 0x7ff99c0613a0 2021-03-09T10:17:19.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094700 con 0x7ff980051540 2021-03-09T10:17:19.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:19.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:19.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:19.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:19.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:19.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:19.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:19.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:19.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:19.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980010da0 con 0x7ff980051540 2021-03-09T10:17:19.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092730 con 0x7ff980051540 2021-03-09T10:17:19.917 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:19.906+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16430 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:19.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.906+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 secure :-1 s=READY pgs=4938 cs=0 l=1 rev1=1 rx=0x7ff9940b7230 tx=0x7ff99409ef50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:19.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980051540 2021-03-09T10:17:19.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980051540 2021-03-09T10:17:19.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:17:19.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.914+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:19.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.914+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:19.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:19.914+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:24.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:24.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:24.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 secure :-1 s=READY pgs=4938 cs=0 l=1 rev1=1 rx=0x7ff9940b7230 tx=0x7ff99409ef50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:24.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 secure :-1 s=READY pgs=4938 cs=0 l=1 rev1=1 rx=0x7ff9940b7230 tx=0x7ff99409ef50).stop 2021-03-09T10:17:24.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:24.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092db0 unknown :-1 s=CLOSED pgs=4938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:24.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:24.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:24.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:24.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052310 con 0x7ff99c0adc20 2021-03-09T10:17:24.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0613a0 2021-03-09T10:17:24.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094700 con 0x7ff980050140 2021-03-09T10:17:24.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:24.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:24.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:24.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:24.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:24.919 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:24.910+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16469 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:24.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:24.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:24.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:24.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:24.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800928d0 con 0x7ff980050140 2021-03-09T10:17:24.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.910+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980033760 con 0x7ff980050140 2021-03-09T10:17:24.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056c10 secure :-1 s=READY pgs=4951 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c08def0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:24.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075080 con 0x7ff980050140 2021-03-09T10:17:24.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08f130 con 0x7ff980050140 2021-03-09T10:17:24.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c022dd0 con 0x7ff980050140 2021-03-09T10:17:24.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.918+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:24.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.922+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:24.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:24.922+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:29.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:29.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:29.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056c10 secure :-1 s=READY pgs=4951 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c08def0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:29.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056c10 secure :-1 s=READY pgs=4951 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c08def0).stop 2021-03-09T10:17:29.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:29.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056c10 unknown :-1 s=CLOSED pgs=4951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:29.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:29.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:29.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:29.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d1f0 con 0x7ff99c0adc20 2021-03-09T10:17:29.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008620 con 0x7ff99c0613a0 2021-03-09T10:17:29.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c820 con 0x7ff980051540 2021-03-09T10:17:29.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:29.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:29.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:29.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:29.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:29.922 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:29.914+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16504 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:29.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:29.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:29.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:29.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:29.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026600 con 0x7ff980051540 2021-03-09T10:17:29.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800928d0 con 0x7ff980051540 2021-03-09T10:17:29.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037f10 secure :-1 s=READY pgs=4961 cs=0 l=1 rev1=1 rx=0x7ff98c006000 tx=0x7ff98c003ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:29.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980051540 2021-03-09T10:17:29.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08f130 con 0x7ff980051540 2021-03-09T10:17:29.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff980051540 2021-03-09T10:17:29.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.922+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:29.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.922+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:29.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:29.922+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:30.705 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:17:30.708 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:17:34.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:34.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:34.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037f10 secure :-1 s=READY pgs=4961 cs=0 l=1 rev1=1 rx=0x7ff98c006000 tx=0x7ff98c003ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:34.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037f10 secure :-1 s=READY pgs=4961 cs=0 l=1 rev1=1 rx=0x7ff98c006000 tx=0x7ff98c003ef0).stop 2021-03-09T10:17:34.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:34.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037f10 unknown :-1 s=CLOSED pgs=4961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:34.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:34.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:34.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:34.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800087c0 con 0x7ff99c0adc20 2021-03-09T10:17:34.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c820 con 0x7ff99c0613a0 2021-03-09T10:17:34.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092a70 con 0x7ff980050140 2021-03-09T10:17:34.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:34.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:34.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:34.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:34.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:34.926 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:34.918+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16543 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:34.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:34.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:34.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:34.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:34.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800359e0 con 0x7ff980050140 2021-03-09T10:17:34.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980010da0 con 0x7ff980050140 2021-03-09T10:17:34.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.918+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b60 secure :-1 s=READY pgs=4971 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994009d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:34.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980050140 2021-03-09T10:17:34.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994099260 con 0x7ff980050140 2021-03-09T10:17:34.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:17:34.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.926+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:34.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.926+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:34.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:34.926+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:39.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:17:39.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:39.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b60 secure :-1 s=READY pgs=4971 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994009d80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:39.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b60 secure :-1 s=READY pgs=4971 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994009d80).stop 2021-03-09T10:17:39.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056b60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:39.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b60 unknown :-1 s=CLOSED pgs=4971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:39.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:39.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:39.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:39.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092a70 con 0x7ff99c0adc20 2021-03-09T10:17:39.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a1c0 con 0x7ff99c0613a0 2021-03-09T10:17:39.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a300 con 0x7ff980051540 2021-03-09T10:17:39.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:39.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:39.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:39.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:39.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:39.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:39.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:39.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:39.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:39.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980010d20 con 0x7ff980051540 2021-03-09T10:17:39.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800359e0 con 0x7ff980051540 2021-03-09T10:17:39.931 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:39.922+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16578 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:39.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057050 secure :-1 s=READY pgs=4980 cs=0 l=1 rev1=1 rx=0x7ff980026970 tx=0x7ff994069f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:39.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069070 con 0x7ff980051540 2021-03-09T10:17:39.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098e90 con 0x7ff980051540 2021-03-09T10:17:39.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:17:39.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.930+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:39.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.930+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:39.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:39.930+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:44.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980057050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:44.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980057050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:44.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057050 secure :-1 s=READY pgs=4980 cs=0 l=1 rev1=1 rx=0x7ff980026970 tx=0x7ff994069f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:44.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057050 secure :-1 s=READY pgs=4980 cs=0 l=1 rev1=1 rx=0x7ff980026970 tx=0x7ff994069f10).stop 2021-03-09T10:17:44.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980057050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:44.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057050 unknown :-1 s=CLOSED pgs=4980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:44.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:44.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:44.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:44.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003ac10 con 0x7ff99c0adc20 2021-03-09T10:17:44.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092a70 con 0x7ff99c0613a0 2021-03-09T10:17:44.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800359e0 con 0x7ff980050140 2021-03-09T10:17:44.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:44.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:44.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:44.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:44.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.922+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:44.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:44.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:44.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:44.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:44.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035b20 con 0x7ff980050140 2021-03-09T10:17:44.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980010d20 con 0x7ff980050140 2021-03-09T10:17:44.935 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:44.926+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16617 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:44.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b10 secure :-1 s=READY pgs=4992 cs=0 l=1 rev1=1 rx=0x7ff980088a10 tx=0x7ff99403eef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:44.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069040 con 0x7ff980050140 2021-03-09T10:17:44.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400bed0 con 0x7ff980050140 2021-03-09T10:17:44.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af8a0 con 0x7ff980050140 2021-03-09T10:17:44.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.934+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:44.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.934+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:44.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:44.934+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:49.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:49.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:49.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b10 secure :-1 s=READY pgs=4992 cs=0 l=1 rev1=1 rx=0x7ff980088a10 tx=0x7ff99403eef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:49.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b10 secure :-1 s=READY pgs=4992 cs=0 l=1 rev1=1 rx=0x7ff980088a10 tx=0x7ff99403eef0).stop 2021-03-09T10:17:49.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:49.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056b10 unknown :-1 s=CLOSED pgs=4992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:49.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:49.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:49.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:49.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800359e0 con 0x7ff99c0adc20 2021-03-09T10:17:49.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092a70 con 0x7ff99c0613a0 2021-03-09T10:17:49.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037eb0 con 0x7ff980051540 2021-03-09T10:17:49.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:49.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:49.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:49.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:49.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:49.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:49.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:49.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:49.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:49.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980037ff0 con 0x7ff980051540 2021-03-09T10:17:49.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035b20 con 0x7ff980051540 2021-03-09T10:17:49.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057000 secure :-1 s=READY pgs=5001 cs=0 l=1 rev1=1 rx=0x7ff994064bc0 tx=0x7ff99408e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:49.939 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:49.930+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16652 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:49.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069080 con 0x7ff980051540 2021-03-09T10:17:49.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409aed0 con 0x7ff980051540 2021-03-09T10:17:49.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af000 con 0x7ff980051540 2021-03-09T10:17:49.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.934+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:49.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.934+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:49.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:49.934+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:54.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980057000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:54.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980057000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:54.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057000 secure :-1 s=READY pgs=5001 cs=0 l=1 rev1=1 rx=0x7ff994064bc0 tx=0x7ff99408e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:54.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057000 secure :-1 s=READY pgs=5001 cs=0 l=1 rev1=1 rx=0x7ff994064bc0 tx=0x7ff99408e000).stop 2021-03-09T10:17:54.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980057000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:54.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980057000 unknown :-1 s=CLOSED pgs=5001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:54.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:54.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:54.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:54.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800379f0 con 0x7ff99c0adc20 2021-03-09T10:17:54.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800099f0 con 0x7ff99c0613a0 2021-03-09T10:17:54.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:54.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800559e0 con 0x7ff980050140 2021-03-09T10:17:54.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:54.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:54.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:54.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:54.940 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:54.930+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16691 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:54.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:54.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:54.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:54.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:54.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092a70 con 0x7ff980050140 2021-03-09T10:17:54.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980037ff0 con 0x7ff980050140 2021-03-09T10:17:54.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a70 secure :-1 s=READY pgs=5016 cs=0 l=1 rev1=1 rx=0x7ff9940142e0 tx=0x7ff99400bf80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:54.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408dd40 con 0x7ff980050140 2021-03-09T10:17:54.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408dd40 con 0x7ff980050140 2021-03-09T10:17:54.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af8a0 con 0x7ff980050140 2021-03-09T10:17:54.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.942+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:54.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.942+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:54.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:54.942+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:17:59.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:59.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:59.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a70 secure :-1 s=READY pgs=5016 cs=0 l=1 rev1=1 rx=0x7ff9940142e0 tx=0x7ff99400bf80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:59.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a70 secure :-1 s=READY pgs=5016 cs=0 l=1 rev1=1 rx=0x7ff9940142e0 tx=0x7ff99400bf80).stop 2021-03-09T10:17:59.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:59.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a70 unknown :-1 s=CLOSED pgs=5016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:59.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:59.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:59.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:59.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057450 con 0x7ff99c0adc20 2021-03-09T10:17:59.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800099a0 con 0x7ff99c0613a0 2021-03-09T10:17:59.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053500 con 0x7ff980051540 2021-03-09T10:17:59.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:17:59.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:59.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:17:59.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:59.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:59.944 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:17:59.934+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16725 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:17:59.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:59.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:59.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:59.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:59.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980053640 con 0x7ff980051540 2021-03-09T10:17:59.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092a70 con 0x7ff980051540 2021-03-09T10:17:59.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056f60 secure :-1 s=READY pgs=5029 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994068130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:59.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff980051540 2021-03-09T10:17:59.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff980051540 2021-03-09T10:17:59.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af000 con 0x7ff980051540 2021-03-09T10:17:59.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.946+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:59.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.946+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:59.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:17:59.946+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:00.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:18:00.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:18:04.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:04.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:04.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056f60 secure :-1 s=READY pgs=5029 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994068130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:04.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056f60 secure :-1 s=READY pgs=5029 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994068130).stop 2021-03-09T10:18:04.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056f60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:04.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056f60 unknown :-1 s=CLOSED pgs=5029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:04.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:04.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:04.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:04.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092ac0 con 0x7ff99c0adc20 2021-03-09T10:18:04.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800359e0 con 0x7ff99c0613a0 2021-03-09T10:18:04.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053360 con 0x7ff980050140 2021-03-09T10:18:04.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:04.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:04.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:04.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:04.947 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:04.938+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16765 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:04.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:04.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.942+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:04.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.942+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:04.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.942+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:04.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.942+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:04.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.942+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027340 con 0x7ff980050140 2021-03-09T10:18:04.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.942+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980053640 con 0x7ff980050140 2021-03-09T10:18:04.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.942+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a00 secure :-1 s=READY pgs=5035 cs=0 l=1 rev1=1 rx=0x7ff9940649b0 tx=0x7ff994016340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:04.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980050140 2021-03-09T10:18:04.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980050140 2021-03-09T10:18:04.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af8a0 con 0x7ff980050140 2021-03-09T10:18:04.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.946+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:04.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.946+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:04.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:04.946+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:09.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:09.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:09.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a00 secure :-1 s=READY pgs=5035 cs=0 l=1 rev1=1 rx=0x7ff9940649b0 tx=0x7ff994016340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:09.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a00 secure :-1 s=READY pgs=5035 cs=0 l=1 rev1=1 rx=0x7ff9940649b0 tx=0x7ff994016340).stop 2021-03-09T10:18:09.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056a00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:09.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056a00 unknown :-1 s=CLOSED pgs=5035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:09.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:09.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:09.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:09.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800553c0 con 0x7ff99c0adc20 2021-03-09T10:18:09.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005490 con 0x7ff99c0613a0 2021-03-09T10:18:09.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a1c0 con 0x7ff980051540 2021-03-09T10:18:09.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:09.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:09.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:09.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:09.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:09.950 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:09.942+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16800 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:09.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.942+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:09.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.942+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:09.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.942+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:09.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.942+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:09.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.942+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001a300 con 0x7ff980051540 2021-03-09T10:18:09.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.942+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027340 con 0x7ff980051540 2021-03-09T10:18:09.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.946+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056ef0 secure :-1 s=READY pgs=5048 cs=0 l=1 rev1=1 rx=0x7ff994078100 tx=0x7ff994036960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:09.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994094360 con 0x7ff980051540 2021-03-09T10:18:09.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994094130 con 0x7ff980051540 2021-03-09T10:18:09.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af000 con 0x7ff980051540 2021-03-09T10:18:09.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.950+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:09.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.954+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:09.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:09.954+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:14.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:14.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:14.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056ef0 secure :-1 s=READY pgs=5048 cs=0 l=1 rev1=1 rx=0x7ff994078100 tx=0x7ff994036960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:14.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056ef0 secure :-1 s=READY pgs=5048 cs=0 l=1 rev1=1 rx=0x7ff994078100 tx=0x7ff994036960).stop 2021-03-09T10:18:14.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:14.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056ef0 unknown :-1 s=CLOSED pgs=5048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:14.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:14.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:14.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:14.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800274e0 con 0x7ff99c0adc20 2021-03-09T10:18:14.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055370 con 0x7ff99c0613a0 2021-03-09T10:18:14.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800359e0 con 0x7ff980050140 2021-03-09T10:18:14.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:14.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:14.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:14.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:14.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:14.954 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:14.946+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16839 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:14.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:14.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:14.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:14.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:14.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094b20 con 0x7ff980050140 2021-03-09T10:18:14.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026600 con 0x7ff980050140 2021-03-09T10:18:14.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056960 secure :-1 s=READY pgs=5055 cs=0 l=1 rev1=1 rx=0x7ff98008c9c0 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:14.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980050140 2021-03-09T10:18:14.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c052130 con 0x7ff980050140 2021-03-09T10:18:14.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017ce0 con 0x7ff980050140 2021-03-09T10:18:14.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.950+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:14.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.954+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:14.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:14.954+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:19.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:19.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:19.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056960 secure :-1 s=READY pgs=5055 cs=0 l=1 rev1=1 rx=0x7ff98008c9c0 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:19.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056960 secure :-1 s=READY pgs=5055 cs=0 l=1 rev1=1 rx=0x7ff98008c9c0 tx=0x7ff98c003e80).stop 2021-03-09T10:18:19.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:19.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056960 unknown :-1 s=CLOSED pgs=5055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:19.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:19.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:19.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:19.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011ef0 con 0x7ff99c0adc20 2021-03-09T10:18:19.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800890f0 con 0x7ff99c0613a0 2021-03-09T10:18:19.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980051e00 con 0x7ff980051540 2021-03-09T10:18:19.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:19.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:19.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:19.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:19.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:19.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:19.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:19.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:19.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:19.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800274e0 con 0x7ff980051540 2021-03-09T10:18:19.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001a300 con 0x7ff980051540 2021-03-09T10:18:19.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056e50 secure :-1 s=READY pgs=5065 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994094130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:19.960 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:19.950+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16874 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:19.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994086900 con 0x7ff980051540 2021-03-09T10:18:19.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980051540 2021-03-09T10:18:19.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af8a0 con 0x7ff980051540 2021-03-09T10:18:19.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.954+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:19.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.954+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:19.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:19.954+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:24.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:24.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:24.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056e50 secure :-1 s=READY pgs=5065 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994094130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:24.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056e50 secure :-1 s=READY pgs=5065 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994094130).stop 2021-03-09T10:18:24.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:24.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056e50 unknown :-1 s=CLOSED pgs=5065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:24.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:24.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:24.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:24.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011ef0 con 0x7ff99c0adc20 2021-03-09T10:18:24.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056b10 con 0x7ff99c0613a0 2021-03-09T10:18:24.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056c50 con 0x7ff980050140 2021-03-09T10:18:24.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:24.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:24.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:24.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:24.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:24.960 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:24.954+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16913 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:24.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:24.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:24.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:24.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:24.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026600 con 0x7ff980050140 2021-03-09T10:18:24.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094b20 con 0x7ff980050140 2021-03-09T10:18:24.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.954+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056620 secure :-1 s=READY pgs=5079 cs=0 l=1 rev1=1 rx=0x7ff980052e60 tx=0x7ff98c00baf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:24.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:18:24.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c052130 con 0x7ff980050140 2021-03-09T10:18:24.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980050140 2021-03-09T10:18:24.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.962+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:24.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.962+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:24.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:24.962+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:29.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:29.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:29.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056620 secure :-1 s=READY pgs=5079 cs=0 l=1 rev1=1 rx=0x7ff980052e60 tx=0x7ff98c00baf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:29.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056620 secure :-1 s=READY pgs=5079 cs=0 l=1 rev1=1 rx=0x7ff980052e60 tx=0x7ff98c00baf0).stop 2021-03-09T10:18:29.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980056620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:29.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980056620 unknown :-1 s=CLOSED pgs=5079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:29.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:29.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:29.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:29.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003be0 con 0x7ff99c0adc20 2021-03-09T10:18:29.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800263c0 con 0x7ff99c0613a0 2021-03-09T10:18:29.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094ab0 con 0x7ff980051540 2021-03-09T10:18:29.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:29.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:29.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:29.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:29.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:29.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:29.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:29.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:29.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:29.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055370 con 0x7ff980051540 2021-03-09T10:18:29.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800274e0 con 0x7ff980051540 2021-03-09T10:18:29.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 secure :-1 s=READY pgs=5087 cs=0 l=1 rev1=1 rx=0x7ff994076010 tx=0x7ff99402adb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:29.967 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:29.954+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16945 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:29.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940a9000 con 0x7ff980051540 2021-03-09T10:18:29.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99406d1f0 con 0x7ff980051540 2021-03-09T10:18:29.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:18:29.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.962+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:29.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.962+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:29.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:29.962+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:30.817 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:18:30.834 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:18:34.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.953+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:34.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.953+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:34.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 secure :-1 s=READY pgs=5087 cs=0 l=1 rev1=1 rx=0x7ff994076010 tx=0x7ff99402adb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:34.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 secure :-1 s=READY pgs=5087 cs=0 l=1 rev1=1 rx=0x7ff994076010 tx=0x7ff99402adb0).stop 2021-03-09T10:18:34.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:34.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 unknown :-1 s=CLOSED pgs=5087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:34.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:34.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:34.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980043aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:34.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980050c90 con 0x7ff99c0adc20 2021-03-09T10:18:34.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056560 con 0x7ff99c0613a0 2021-03-09T10:18:34.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800566a0 con 0x7ff980050140 2021-03-09T10:18:34.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:34.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:34.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:34.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:34.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980043aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:34.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:34.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:34.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:34.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:34.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800567e0 con 0x7ff980050140 2021-03-09T10:18:34.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.957+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055370 con 0x7ff980050140 2021-03-09T10:18:34.970 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:34.957+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16987 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:34.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980043aa0 secure :-1 s=READY pgs=5099 cs=0 l=1 rev1=1 rx=0x7ff994075fe0 tx=0x7ff994037720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:34.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.961+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99406d1f0 con 0x7ff980050140 2021-03-09T10:18:34.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.965+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99406d1f0 con 0x7ff980050140 2021-03-09T10:18:34.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.965+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:18:34.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.965+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:34.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.969+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:34.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:34.969+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:39.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980043aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:39.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980043aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:39.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980043aa0 secure :-1 s=READY pgs=5099 cs=0 l=1 rev1=1 rx=0x7ff994075fe0 tx=0x7ff994037720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:39.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980043aa0 secure :-1 s=READY pgs=5099 cs=0 l=1 rev1=1 rx=0x7ff994075fe0 tx=0x7ff994037720).stop 2021-03-09T10:18:39.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980043aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:39.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980043aa0 unknown :-1 s=CLOSED pgs=5099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:39.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:39.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:39.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:39.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800553c0 con 0x7ff99c0adc20 2021-03-09T10:18:39.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026320 con 0x7ff99c0613a0 2021-03-09T10:18:39.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008b790 con 0x7ff980051540 2021-03-09T10:18:39.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:39.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:39.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:39.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:39.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:39.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:39.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:39.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:39.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:39.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094a60 con 0x7ff980051540 2021-03-09T10:18:39.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800567e0 con 0x7ff980051540 2021-03-09T10:18:39.972 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:39.961+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17022 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:39.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.961+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 secure :-1 s=READY pgs=5108 cs=0 l=1 rev1=1 rx=0x7ff99406ffd0 tx=0x7ff994037c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:39.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.965+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b1560 con 0x7ff980051540 2021-03-09T10:18:39.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.965+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940b11c0 con 0x7ff980051540 2021-03-09T10:18:39.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:18:39.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.969+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:39.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.969+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:39.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:39.969+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:44.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:18:44.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:44.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 secure :-1 s=READY pgs=5108 cs=0 l=1 rev1=1 rx=0x7ff99406ffd0 tx=0x7ff994037c00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:44.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 secure :-1 s=READY pgs=5108 cs=0 l=1 rev1=1 rx=0x7ff99406ffd0 tx=0x7ff994037c00).stop 2021-03-09T10:18:44.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980056d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:44.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980056d50 unknown :-1 s=CLOSED pgs=5108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:44.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:44.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:44.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:44.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800553c0 con 0x7ff99c0adc20 2021-03-09T10:18:44.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800922e0 con 0x7ff99c0613a0 2021-03-09T10:18:44.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800438b0 con 0x7ff980050140 2021-03-09T10:18:44.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:44.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:44.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:44.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:44.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:44.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:44.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:44.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:44.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:44.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980043a20 con 0x7ff980050140 2021-03-09T10:18:44.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094a60 con 0x7ff980050140 2021-03-09T10:18:44.976 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:44.965+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17061 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:44.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.965+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b5f0 secure :-1 s=READY pgs=5119 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940a9140).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:44.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b11c0 con 0x7ff980050140 2021-03-09T10:18:44.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980050140 2021-03-09T10:18:44.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:18:44.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.973+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:44.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.973+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:44.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:44.973+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:49.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98008b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:18:49.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98008b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:49.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b5f0 secure :-1 s=READY pgs=5119 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940a9140).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:49.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b5f0 secure :-1 s=READY pgs=5119 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940a9140).stop 2021-03-09T10:18:49.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98008b5f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:49.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b5f0 unknown :-1 s=CLOSED pgs=5119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:49.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:49.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:49.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:49.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800553c0 con 0x7ff99c0adc20 2021-03-09T10:18:49.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092480 con 0x7ff99c0613a0 2021-03-09T10:18:49.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043890 con 0x7ff980051540 2021-03-09T10:18:49.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:49.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:49.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:49.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:49.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:49.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:49.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:49.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:49.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:49.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026130 con 0x7ff980051540 2021-03-09T10:18:49.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026600 con 0x7ff980051540 2021-03-09T10:18:49.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.969+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043bb0 secure :-1 s=READY pgs=5129 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:49.980 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:49.969+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17096 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:49.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075020 con 0x7ff980051540 2021-03-09T10:18:49.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:18:49.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c02c140 con 0x7ff980051540 2021-03-09T10:18:49.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.973+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:49.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.977+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:49.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:49.977+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:54.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980043bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:54.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980043bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:54.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043bb0 secure :-1 s=READY pgs=5129 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:54.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043bb0 secure :-1 s=READY pgs=5129 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00bfd0).stop 2021-03-09T10:18:54.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980043bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:54.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980043bb0 unknown :-1 s=CLOSED pgs=5129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:54.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:54.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:54.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:54.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c3b0 con 0x7ff99c0adc20 2021-03-09T10:18:54.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026e90 con 0x7ff99c0613a0 2021-03-09T10:18:54.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092480 con 0x7ff980050140 2021-03-09T10:18:54.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:54.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:54.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:54.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:54.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:54.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:54.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:54.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:54.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:54.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026600 con 0x7ff980050140 2021-03-09T10:18:54.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980043a20 con 0x7ff980050140 2021-03-09T10:18:54.983 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:54.973+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17135 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:54.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b400 secure :-1 s=READY pgs=5140 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff99408d1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:54.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980050140 2021-03-09T10:18:54.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994037790 con 0x7ff980050140 2021-03-09T10:18:54.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:18:54.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.977+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:54.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.981+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:54.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:54.981+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:18:59.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98008b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:59.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98008b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:59.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b400 secure :-1 s=READY pgs=5140 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff99408d1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:59.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b400 secure :-1 s=READY pgs=5140 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff99408d1f0).stop 2021-03-09T10:18:59.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98008b400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:59.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98008b400 unknown :-1 s=CLOSED pgs=5140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:59.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:59.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:59.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:59.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092c50 con 0x7ff99c0adc20 2021-03-09T10:18:59.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049670 con 0x7ff99c0613a0 2021-03-09T10:18:59.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026e90 con 0x7ff980051540 2021-03-09T10:18:59.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:18:59.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:59.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:18:59.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:59.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.977+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:59.987 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:18:59.981+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17170 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:18:59.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.981+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:59.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.981+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:59.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.981+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:59.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.981+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:59.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.981+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008c3b0 con 0x7ff980051540 2021-03-09T10:18:59.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.981+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026600 con 0x7ff980051540 2021-03-09T10:18:59.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 rx=0x7ff994093300 tx=0x7ff994099070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:59.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994084d40 con 0x7ff980051540 2021-03-09T10:18:59.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994084d40 con 0x7ff980051540 2021-03-09T10:18:59.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:18:59.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.993+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:59.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.993+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:59.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:18:59.993+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:00.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:19:00.885 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:19:04.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:04.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:04.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 rx=0x7ff994093300 tx=0x7ff994099070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:04.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 rx=0x7ff994093300 tx=0x7ff994099070).stop 2021-03-09T10:19:04.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:04.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 unknown :-1 s=CLOSED pgs=5155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:04.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:04.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800384e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:04.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:04.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092c50 con 0x7ff99c0adc20 2021-03-09T10:19:04.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f390 con 0x7ff99c0613a0 2021-03-09T10:19:04.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035fb0 con 0x7ff980050140 2021-03-09T10:19:04.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:04.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:04.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:04.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:04.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.981+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800384e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:04.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.985+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:04.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.985+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:04.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.985+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:04.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.985+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:04.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.985+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008ba70 con 0x7ff980050140 2021-03-09T10:19:04.999 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:04.985+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17205 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:05.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.985+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026130 con 0x7ff980050140 2021-03-09T10:19:05.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.985+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800384e0 secure :-1 s=READY pgs=5161 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c0797b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:05.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075080 con 0x7ff980050140 2021-03-09T10:19:05.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980050140 2021-03-09T10:19:05.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c07d0f0 con 0x7ff980050140 2021-03-09T10:19:05.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.993+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:05.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.993+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:05.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:04.993+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:09.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800384e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:09.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800384e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:09.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800384e0 secure :-1 s=READY pgs=5161 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c0797b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:09.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800384e0 secure :-1 s=READY pgs=5161 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c0797b0).stop 2021-03-09T10:19:09.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800384e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:09.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800384e0 unknown :-1 s=CLOSED pgs=5161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:09.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:09.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:09.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:09.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092c50 con 0x7ff99c0adc20 2021-03-09T10:19:09.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035fb0 con 0x7ff99c0613a0 2021-03-09T10:19:09.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026e90 con 0x7ff980051540 2021-03-09T10:19:09.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:09.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:09.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:09.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:09.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:09.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:09.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:09.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:09.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:09.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003f390 con 0x7ff980051540 2021-03-09T10:19:09.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008c3b0 con 0x7ff980051540 2021-03-09T10:19:09.995 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:09.989+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17244 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:09.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.989+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 secure :-1 s=READY pgs=5172 cs=0 l=1 rev1=1 rx=0x7ff9800394e0 tx=0x7ff994098ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:09.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d3b0 con 0x7ff980051540 2021-03-09T10:19:09.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980051540 2021-03-09T10:19:10.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:19:10.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.997+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:10.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.997+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:10.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:09.997+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:14.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:14.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:14.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 secure :-1 s=READY pgs=5172 cs=0 l=1 rev1=1 rx=0x7ff9800394e0 tx=0x7ff994098ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:14.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 secure :-1 s=READY pgs=5172 cs=0 l=1 rev1=1 rx=0x7ff9800394e0 tx=0x7ff994098ee0).stop 2021-03-09T10:19:14.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980037ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:14.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980037ff0 unknown :-1 s=CLOSED pgs=5172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:14.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:14.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:14.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980010cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:14.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cce0 con 0x7ff99c0adc20 2021-03-09T10:19:14.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026e90 con 0x7ff99c0613a0 2021-03-09T10:19:14.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f470 con 0x7ff980050140 2021-03-09T10:19:14.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:14.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:14.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:14.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:14.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980010cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:14.997 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:14.989+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17282 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:14.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:14.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.989+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:14.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:14.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:14.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008c3b0 con 0x7ff980050140 2021-03-09T10:19:14.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003f390 con 0x7ff980050140 2021-03-09T10:19:14.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980010cd0 secure :-1 s=READY pgs=5183 cs=0 l=1 rev1=1 rx=0x7ff994075500 tx=0x7ff994009e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:15.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980050140 2021-03-09T10:19:15.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:14.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980050140 2021-03-09T10:19:15.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:15.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:19:15.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:15.001+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:15.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:15.001+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:15.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:15.001+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:19.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980010cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:19:19.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980010cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:19.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980010cd0 secure :-1 s=READY pgs=5183 cs=0 l=1 rev1=1 rx=0x7ff994075500 tx=0x7ff994009e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:19.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980010cd0 secure :-1 s=READY pgs=5183 cs=0 l=1 rev1=1 rx=0x7ff994075500 tx=0x7ff994009e70).stop 2021-03-09T10:19:19.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980010cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:19.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980010cd0 unknown :-1 s=CLOSED pgs=5183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:19.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:19.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:19.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:19.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f470 con 0x7ff99c0adc20 2021-03-09T10:19:19.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cce0 con 0x7ff99c0613a0 2021-03-09T10:19:19.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800440d0 con 0x7ff980051540 2021-03-09T10:19:19.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:19.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:19.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:19.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:19.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:20.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:20.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:20.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:20.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:20.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092c00 con 0x7ff980051540 2021-03-09T10:19:20.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008c3b0 con 0x7ff980051540 2021-03-09T10:19:20.001 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:19.993+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17318 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:20.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:19.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5193 cs=0 l=1 rev1=1 rx=0x7ff9940752d0 tx=0x7ff9940aa0f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:20.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:20.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:19:20.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:20.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:19:20.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:20.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:19:20.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:20.005+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:20.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:20.005+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:20.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:20.005+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:24.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:24.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:25.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5193 cs=0 l=1 rev1=1 rx=0x7ff9940752d0 tx=0x7ff9940aa0f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:25.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5193 cs=0 l=1 rev1=1 rx=0x7ff9940752d0 tx=0x7ff9940aa0f0).stop 2021-03-09T10:19:25.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:25.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:25.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:25.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:25.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:25.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a250 con 0x7ff99c0adc20 2021-03-09T10:19:25.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800459d0 con 0x7ff99c0613a0 2021-03-09T10:19:25.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003d820 con 0x7ff980050140 2021-03-09T10:19:25.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:25.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:25.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:25.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:25.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:25.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:25.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:25.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:25.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:25.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008c3b0 con 0x7ff980050140 2021-03-09T10:19:25.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:24.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092c00 con 0x7ff980050140 2021-03-09T10:19:25.006 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:24.997+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17356 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:25.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:25.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 secure :-1 s=READY pgs=5204 cs=0 l=1 rev1=1 rx=0x7ff994087040 tx=0x7ff99402a200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:25.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:25.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994092860 con 0x7ff980050140 2021-03-09T10:19:25.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:25.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994092860 con 0x7ff980050140 2021-03-09T10:19:25.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:25.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:19:25.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:25.009+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:25.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:25.009+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:25.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:25.009+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:30.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:30.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:30.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 secure :-1 s=READY pgs=5204 cs=0 l=1 rev1=1 rx=0x7ff994087040 tx=0x7ff99402a200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:30.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 secure :-1 s=READY pgs=5204 cs=0 l=1 rev1=1 rx=0x7ff994087040 tx=0x7ff99402a200).stop 2021-03-09T10:19:30.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:30.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 unknown :-1 s=CLOSED pgs=5204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:30.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:30.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:30.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:30.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055e10 con 0x7ff99c0adc20 2021-03-09T10:19:30.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045b70 con 0x7ff99c0613a0 2021-03-09T10:19:30.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:29.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f470 con 0x7ff980051540 2021-03-09T10:19:30.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.001+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:30.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.001+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:30.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:30.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:30.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:30.008 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:30.001+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17389 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:30.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.005+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:30.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.005+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:30.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.005+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:30.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.005+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:30.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.005+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000a200 con 0x7ff980051540 2021-03-09T10:19:30.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.005+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008c3b0 con 0x7ff980051540 2021-03-09T10:19:30.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.005+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5219 cs=0 l=1 rev1=1 rx=0x7ff99400bfa0 tx=0x7ff994001e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:30.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994092860 con 0x7ff980051540 2021-03-09T10:19:30.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994092860 con 0x7ff980051540 2021-03-09T10:19:30.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:19:30.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.009+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:30.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.013+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:30.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:30.013+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:30.947 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:19:30.950 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:19:35.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:35.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:35.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5219 cs=0 l=1 rev1=1 rx=0x7ff99400bfa0 tx=0x7ff994001e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:35.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5219 cs=0 l=1 rev1=1 rx=0x7ff99400bfa0 tx=0x7ff994001e50).stop 2021-03-09T10:19:35.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:35.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:35.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:35.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:35.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:35.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049830 con 0x7ff99c0adc20 2021-03-09T10:19:35.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f470 con 0x7ff99c0613a0 2021-03-09T10:19:35.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980058550 con 0x7ff980050140 2021-03-09T10:19:35.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:35.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:35.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:35.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:35.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.005+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:35.124 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:35.005+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17430 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:35.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:35.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.005+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:35.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:35.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.005+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:35.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008ce80 con 0x7ff980050140 2021-03-09T10:19:35.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ba70 con 0x7ff980050140 2021-03-09T10:19:35.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 secure :-1 s=READY pgs=5228 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c003f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:35.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075020 con 0x7ff980050140 2021-03-09T10:19:35.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a9ec0 con 0x7ff980050140 2021-03-09T10:19:35.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980050140 2021-03-09T10:19:35.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.017+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:35.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.017+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:35.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:35.017+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:40.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:40.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:40.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 secure :-1 s=READY pgs=5228 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c003f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:40.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 secure :-1 s=READY pgs=5228 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c003f60).stop 2021-03-09T10:19:40.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:40.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037ff0 unknown :-1 s=CLOSED pgs=5228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:40.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:40.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:40.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:40.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:40.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f470 con 0x7ff99c0adc20 2021-03-09T10:19:40.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:40.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092480 con 0x7ff99c0613a0 2021-03-09T10:19:40.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c3b0 con 0x7ff980051540 2021-03-09T10:19:40.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:40.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.005+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:40.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:40.015 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:40.009+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17466 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:40.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:40.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:40.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:40.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:40.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045830 con 0x7ff980051540 2021-03-09T10:19:40.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ce80 con 0x7ff980051540 2021-03-09T10:19:40.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5237 cs=0 l=1 rev1=1 rx=0x7ff98c07e000 tx=0x7ff98c00ba80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:40.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.013+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T10:19:40.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.013+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c099120 con 0x7ff980051540 2021-03-09T10:19:40.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:19:40.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.017+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:40.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.017+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:40.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:40.017+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:45.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:45.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:45.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5237 cs=0 l=1 rev1=1 rx=0x7ff98c07e000 tx=0x7ff98c00ba80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:45.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5237 cs=0 l=1 rev1=1 rx=0x7ff98c07e000 tx=0x7ff98c00ba80).stop 2021-03-09T10:19:45.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:45.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:45.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:45.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:45.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:45.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:45.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:45.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092480 con 0x7ff99c0adc20 2021-03-09T10:19:45.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c3b0 con 0x7ff99c0613a0 2021-03-09T10:19:45.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045eb0 con 0x7ff980050140 2021-03-09T10:19:45.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:45.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:45.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:45.017 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:45.013+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17504 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:45.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.013+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:45.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.013+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:45.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.013+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:45.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.013+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:45.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.013+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045ff0 con 0x7ff980050140 2021-03-09T10:19:45.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.013+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045830 con 0x7ff980050140 2021-03-09T10:19:45.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.013+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5249 cs=0 l=1 rev1=1 rx=0x7ff98008b9a0 tx=0x7ff98c07e150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:45.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:19:45.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980050140 2021-03-09T10:19:45.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.021+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980050140 2021-03-09T10:19:45.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.021+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:45.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.021+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:45.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:45.021+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:50.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:50.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:50.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5249 cs=0 l=1 rev1=1 rx=0x7ff98008b9a0 tx=0x7ff98c07e150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:50.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5249 cs=0 l=1 rev1=1 rx=0x7ff98008b9a0 tx=0x7ff98c07e150).stop 2021-03-09T10:19:50.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:50.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=CLOSED pgs=5249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:50.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:50.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:50.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:50.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049810 con 0x7ff99c0adc20 2021-03-09T10:19:50.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092480 con 0x7ff99c0613a0 2021-03-09T10:19:50.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c3b0 con 0x7ff980051540 2021-03-09T10:19:50.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:50.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:50.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:50.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:50.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.012+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:50.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:50.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:50.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:50.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:50.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045f50 con 0x7ff980051540 2021-03-09T10:19:50.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000a200 con 0x7ff980051540 2021-03-09T10:19:50.024 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:50.016+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17540 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:50.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5257 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99400bed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:50.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994075110 con 0x7ff980051540 2021-03-09T10:19:50.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:19:50.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:19:50.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.020+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:50.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:50.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:50.024+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:19:55.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:55.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:55.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5257 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99400bed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:55.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5257 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99400bed0).stop 2021-03-09T10:19:55.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:55.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:55.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:55.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:55.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:55.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092480 con 0x7ff99c0adc20 2021-03-09T10:19:55.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e50 con 0x7ff99c0613a0 2021-03-09T10:19:55.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056d50 con 0x7ff980050140 2021-03-09T10:19:55.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:19:55.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:55.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:19:55.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:55.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.016+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:55.025 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:19:55.020+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17577 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:19:55.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:55.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:55.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:55.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:55.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980056e90 con 0x7ff980050140 2021-03-09T10:19:55.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045f50 con 0x7ff980050140 2021-03-09T10:19:55.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5273 cs=0 l=1 rev1=1 rx=0x7ff994036010 tx=0x7ff994081670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:55.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940af000 con 0x7ff980050140 2021-03-09T10:19:55.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940af160 con 0x7ff980050140 2021-03-09T10:19:55.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:19:55.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.028+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:55.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.028+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:55.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:19:55.028+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:00.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:00.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:00.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5273 cs=0 l=1 rev1=1 rx=0x7ff994036010 tx=0x7ff994081670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:00.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5273 cs=0 l=1 rev1=1 rx=0x7ff994036010 tx=0x7ff994081670).stop 2021-03-09T10:20:00.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:00.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=CLOSED pgs=5273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:00.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:00.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:00.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:00.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008a00 con 0x7ff99c0adc20 2021-03-09T10:20:00.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045f50 con 0x7ff99c0613a0 2021-03-09T10:20:00.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056d00 con 0x7ff980051540 2021-03-09T10:20:00.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:00.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:00.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:00.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:00.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.020+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:00.029 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:00.020+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17614 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:00.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:00.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:00.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:00.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:00.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800922e0 con 0x7ff980051540 2021-03-09T10:20:00.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980056e90 con 0x7ff980051540 2021-03-09T10:20:00.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5283 cs=0 l=1 rev1=1 rx=0x7ff980037ed0 tx=0x7ff9940aa160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:00.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940afa30 con 0x7ff980051540 2021-03-09T10:20:00.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940afa30 con 0x7ff980051540 2021-03-09T10:20:00.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af000 con 0x7ff980051540 2021-03-09T10:20:00.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.032+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:00.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.032+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:00.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:00.032+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:01.029 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:20:01.032 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:20:05.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:05.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:05.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5283 cs=0 l=1 rev1=1 rx=0x7ff980037ed0 tx=0x7ff9940aa160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:05.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5283 cs=0 l=1 rev1=1 rx=0x7ff980037ed0 tx=0x7ff9940aa160).stop 2021-03-09T10:20:05.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:05.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:05.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:05.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:05.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:05.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045f50 con 0x7ff99c0adc20 2021-03-09T10:20:05.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980008a00 con 0x7ff99c0613a0 2021-03-09T10:20:05.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020b10 con 0x7ff980050140 2021-03-09T10:20:05.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:05.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:05.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:05.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:05.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:05.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:05.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:05.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:05.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:05.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020c50 con 0x7ff980050140 2021-03-09T10:20:05.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800922e0 con 0x7ff980050140 2021-03-09T10:20:05.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5289 cs=0 l=1 rev1=1 rx=0x7ff9940952e0 tx=0x7ff99405dc00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:05.034 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:05.024+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17648 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:05.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095120 con 0x7ff980050140 2021-03-09T10:20:05.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400c150 con 0x7ff980050140 2021-03-09T10:20:05.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af000 con 0x7ff980050140 2021-03-09T10:20:05.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.032+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:05.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:05.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:05.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:10.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:10.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.024+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:10.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5289 cs=0 l=1 rev1=1 rx=0x7ff9940952e0 tx=0x7ff99405dc00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:10.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.024+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5289 cs=0 l=1 rev1=1 rx=0x7ff9940952e0 tx=0x7ff99405dc00).stop 2021-03-09T10:20:10.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.024+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:10.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.024+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=CLOSED pgs=5289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:10.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:10.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:10.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:10.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020b10 con 0x7ff99c0adc20 2021-03-09T10:20:10.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045f50 con 0x7ff99c0613a0 2021-03-09T10:20:10.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800922e0 con 0x7ff980051540 2021-03-09T10:20:10.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:10.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:10.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:10.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:10.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:10.036 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:10.028+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17688 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:10.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:10.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:10.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:10.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:10.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092420 con 0x7ff980051540 2021-03-09T10:20:10.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020c50 con 0x7ff980051540 2021-03-09T10:20:10.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.032+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5301 cs=0 l=1 rev1=1 rx=0x7ff994095510 tx=0x7ff99405f2a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:10.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980051540 2021-03-09T10:20:10.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980051540 2021-03-09T10:20:10.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:20:10.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.036+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:10.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:10.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:10.036+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:15.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:15.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:15.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5301 cs=0 l=1 rev1=1 rx=0x7ff994095510 tx=0x7ff99405f2a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:15.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5301 cs=0 l=1 rev1=1 rx=0x7ff994095510 tx=0x7ff99405f2a0).stop 2021-03-09T10:20:15.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:15.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:15.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:15.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:15.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:15.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053800 con 0x7ff99c0adc20 2021-03-09T10:20:15.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045f50 con 0x7ff99c0613a0 2021-03-09T10:20:15.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092290 con 0x7ff980050140 2021-03-09T10:20:15.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:15.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:15.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:15.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.028+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:15.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.032+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:15.039 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:15.032+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17722 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:15.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.036+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:15.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.036+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:15.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.036+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:15.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.036+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:15.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.036+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000ecb0 con 0x7ff980050140 2021-03-09T10:20:15.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.036+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045830 con 0x7ff980050140 2021-03-09T10:20:15.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.036+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5315 cs=0 l=1 rev1=1 rx=0x7ff98008b970 tx=0x7ff98c052210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:15.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075020 con 0x7ff980050140 2021-03-09T10:20:15.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980050140 2021-03-09T10:20:15.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c08f130 con 0x7ff980050140 2021-03-09T10:20:15.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.040+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:15.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.040+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:15.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:15.040+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:20.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:20.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:20.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5315 cs=0 l=1 rev1=1 rx=0x7ff98008b970 tx=0x7ff98c052210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:20.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 secure :-1 s=READY pgs=5315 cs=0 l=1 rev1=1 rx=0x7ff98008b970 tx=0x7ff98c052210).stop 2021-03-09T10:20:20.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff9800459c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:20.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff9800459c0 unknown :-1 s=CLOSED pgs=5315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:20.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:20.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:20.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:20.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053800 con 0x7ff99c0adc20 2021-03-09T10:20:20.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:20.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f180 con 0x7ff99c0613a0 2021-03-09T10:20:20.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045f50 con 0x7ff980051540 2021-03-09T10:20:20.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:20.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:20.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:20.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.032+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:20.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.036+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:20.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.036+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:20.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.036+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:20.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.036+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:20.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.036+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092290 con 0x7ff980051540 2021-03-09T10:20:20.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.036+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092420 con 0x7ff980051540 2021-03-09T10:20:20.044 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:20.036+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17762 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:20.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.036+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5322 cs=0 l=1 rev1=1 rx=0x7ff994086560 tx=0x7ff994081d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:20.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980051540 2021-03-09T10:20:20.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980051540 2021-03-09T10:20:20.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:20:20.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.044+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:20.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.044+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:20.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:20.044+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:25.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:25.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:25.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5322 cs=0 l=1 rev1=1 rx=0x7ff994086560 tx=0x7ff994081d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:25.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5322 cs=0 l=1 rev1=1 rx=0x7ff994086560 tx=0x7ff994081d90).stop 2021-03-09T10:20:25.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:25.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:25.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:25.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:25.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:25.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092100 con 0x7ff99c0adc20 2021-03-09T10:20:25.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f330 con 0x7ff99c0613a0 2021-03-09T10:20:25.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027650 con 0x7ff980050140 2021-03-09T10:20:25.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:25.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:25.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:25.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:25.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.036+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:25.045 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:25.040+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17793 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:25.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.040+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:25.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.040+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:25.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.040+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:25.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.040+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:25.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.040+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045830 con 0x7ff980050140 2021-03-09T10:20:25.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.040+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092290 con 0x7ff980050140 2021-03-09T10:20:25.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.040+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ee40 secure :-1 s=READY pgs=5335 cs=0 l=1 rev1=1 rx=0x7ff994062ec0 tx=0x7ff99405fec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:25.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400d1c0 con 0x7ff980050140 2021-03-09T10:20:25.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940aad20 con 0x7ff980050140 2021-03-09T10:20:25.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:20:25.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.048+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:25.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.048+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:25.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:25.048+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:30.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:30.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:30.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ee40 secure :-1 s=READY pgs=5335 cs=0 l=1 rev1=1 rx=0x7ff994062ec0 tx=0x7ff99405fec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:30.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ee40 secure :-1 s=READY pgs=5335 cs=0 l=1 rev1=1 rx=0x7ff994062ec0 tx=0x7ff99405fec0).stop 2021-03-09T10:20:30.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000ee40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:30.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ee40 unknown :-1 s=CLOSED pgs=5335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:30.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:30.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:30.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:30.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f330 con 0x7ff99c0adc20 2021-03-09T10:20:30.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092100 con 0x7ff99c0613a0 2021-03-09T10:20:30.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800925d0 con 0x7ff980051540 2021-03-09T10:20:30.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:30.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:30.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:30.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:30.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.040+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:30.049 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:30.044+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17836 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:30.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:30.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:30.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:30.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:30.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045f50 con 0x7ff980051540 2021-03-09T10:20:30.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000ecb0 con 0x7ff980051540 2021-03-09T10:20:30.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5347 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c017d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:30.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980051540 2021-03-09T10:20:30.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:20:30.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980051540 2021-03-09T10:20:30.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.056+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:30.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.056+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:30.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:30.056+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:31.084 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:20:31.087 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:20:35.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:35.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:35.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5347 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c017d50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:35.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5347 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c017d50).stop 2021-03-09T10:20:35.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:35.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:35.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:35.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:35.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:35.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005280 con 0x7ff99c0adc20 2021-03-09T10:20:35.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092100 con 0x7ff99c0613a0 2021-03-09T10:20:35.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800925d0 con 0x7ff980050140 2021-03-09T10:20:35.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:35.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:35.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:35.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:35.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:35.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:35.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:35.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:35.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:35.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000f100 con 0x7ff980050140 2021-03-09T10:20:35.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.044+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045830 con 0x7ff980050140 2021-03-09T10:20:35.176 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:35.044+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17870 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:35.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.048+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ec10 secure :-1 s=READY pgs=5354 cs=0 l=1 rev1=1 rx=0x7ff9800573e0 tx=0x7ff994011470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:35.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980050140 2021-03-09T10:20:35.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086210 con 0x7ff980050140 2021-03-09T10:20:35.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:20:35.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.052+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:35.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.056+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:35.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:35.056+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:40.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:40.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:40.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ec10 secure :-1 s=READY pgs=5354 cs=0 l=1 rev1=1 rx=0x7ff9800573e0 tx=0x7ff994011470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:40.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ec10 secure :-1 s=READY pgs=5354 cs=0 l=1 rev1=1 rx=0x7ff9800573e0 tx=0x7ff994011470).stop 2021-03-09T10:20:40.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000ec10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:40.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000ec10 unknown :-1 s=CLOSED pgs=5354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:40.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:40.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:40.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:40.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0adc20 2021-03-09T10:20:40.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092100 con 0x7ff99c0613a0 2021-03-09T10:20:40.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004bf60 con 0x7ff980051540 2021-03-09T10:20:40.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:40.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:40.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:40.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:40.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.048+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:40.056 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:40.048+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17910 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:40.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:40.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:40.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:40.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:40.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980037fa0 con 0x7ff980051540 2021-03-09T10:20:40.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000f100 con 0x7ff980051540 2021-03-09T10:20:40.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5366 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994009e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:40.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940ab820 con 0x7ff980051540 2021-03-09T10:20:40.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99402a5e0 con 0x7ff980051540 2021-03-09T10:20:40.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:20:40.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.060+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:40.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.060+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:40.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:40.060+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:45.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:45.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:45.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5366 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994009e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:45.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5366 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994009e60).stop 2021-03-09T10:20:45.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:45.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:45.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:45.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:45.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:45.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0adc20 2021-03-09T10:20:45.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004bf20 con 0x7ff99c0613a0 2021-03-09T10:20:45.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f060 con 0x7ff980050140 2021-03-09T10:20:45.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:45.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:45.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:45.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:45.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:45.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:45.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:45.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:45.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:45.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000f1d0 con 0x7ff980050140 2021-03-09T10:20:45.062 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:45.052+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17944 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:45.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.052+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045f50 con 0x7ff980050140 2021-03-09T10:20:45.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.056+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000eb70 secure :-1 s=READY pgs=5376 cs=0 l=1 rev1=1 rx=0x7ff98c016ab0 tx=0x7ff98c017dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:45.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980050140 2021-03-09T10:20:45.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02c140 con 0x7ff980050140 2021-03-09T10:20:45.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.064+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980050140 2021-03-09T10:20:45.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.064+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:45.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.064+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:45.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:45.064+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:50.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:50.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:50.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000eb70 secure :-1 s=READY pgs=5376 cs=0 l=1 rev1=1 rx=0x7ff98c016ab0 tx=0x7ff98c017dc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:50.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000eb70 secure :-1 s=READY pgs=5376 cs=0 l=1 rev1=1 rx=0x7ff98c016ab0 tx=0x7ff98c017dc0).stop 2021-03-09T10:20:50.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff98000eb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:50.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff98000eb70 unknown :-1 s=CLOSED pgs=5376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:50.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:50.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:50.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:50.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043ac0 con 0x7ff99c0adc20 2021-03-09T10:20:50.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0613a0 2021-03-09T10:20:50.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008b600 con 0x7ff980051540 2021-03-09T10:20:50.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:50.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:50.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:50.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:50.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:50.063 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:50.056+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17984 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:50.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:50.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:50.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:50.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:50.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045f50 con 0x7ff980051540 2021-03-09T10:20:50.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980037fa0 con 0x7ff980051540 2021-03-09T10:20:50.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5387 cs=0 l=1 rev1=1 rx=0x7ff99402a7d0 tx=0x7ff994062ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:50.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:20:50.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:20:50.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.064+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:20:50.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.064+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:50.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.068+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:50.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:50.068+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:20:55.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:55.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:55.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5387 cs=0 l=1 rev1=1 rx=0x7ff99402a7d0 tx=0x7ff994062ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:55.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5387 cs=0 l=1 rev1=1 rx=0x7ff99402a7d0 tx=0x7ff994062ef0).stop 2021-03-09T10:20:55.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:55.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:55.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:55.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:55.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:55.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0adc20 2021-03-09T10:20:55.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d7c0 con 0x7ff99c0613a0 2021-03-09T10:20:55.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008b600 con 0x7ff980050140 2021-03-09T10:20:55.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:20:55.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:55.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:20:55.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:55.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:55.066 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:20:55.060+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18018 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:20:55.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:55.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:55.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:55.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:55.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000f000 con 0x7ff980050140 2021-03-09T10:20:55.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000f1d0 con 0x7ff980050140 2021-03-09T10:20:55.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037fa0 secure :-1 s=READY pgs=5396 cs=0 l=1 rev1=1 rx=0x7ff98c034010 tx=0x7ff98c006650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:55.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.064+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:20:55.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.064+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980050140 2021-03-09T10:20:55.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980050140 2021-03-09T10:20:55.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.068+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:55.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.068+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:55.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:20:55.068+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:00.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:00.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:00.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037fa0 secure :-1 s=READY pgs=5396 cs=0 l=1 rev1=1 rx=0x7ff98c034010 tx=0x7ff98c006650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:00.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037fa0 secure :-1 s=READY pgs=5396 cs=0 l=1 rev1=1 rx=0x7ff98c034010 tx=0x7ff98c006650).stop 2021-03-09T10:21:00.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980037fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:00.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980037fa0 unknown :-1 s=CLOSED pgs=5396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:00.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:00.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:00.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:00.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0adc20 2021-03-09T10:21:00.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d7c0 con 0x7ff99c0613a0 2021-03-09T10:21:00.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088b10 con 0x7ff980051540 2021-03-09T10:21:00.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:00.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:00.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:00.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:00.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:00.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:00.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:00.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:00.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:00.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980038490 con 0x7ff980051540 2021-03-09T10:21:00.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.060+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045f50 con 0x7ff980051540 2021-03-09T10:21:00.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.064+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5407 cs=0 l=1 rev1=1 rx=0x7ff980094700 tx=0x7ff99403e130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:00.071 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:00.064+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18057 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:00.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:21:00.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994012df0 con 0x7ff980051540 2021-03-09T10:21:00.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:21:00.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.068+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:00.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.072+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:00.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:00.072+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:01.131 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:21:01.134 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:21:05.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:05.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:05.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5407 cs=0 l=1 rev1=1 rx=0x7ff980094700 tx=0x7ff99403e130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:05.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5407 cs=0 l=1 rev1=1 rx=0x7ff980094700 tx=0x7ff99403e130).stop 2021-03-09T10:21:05.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:05.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:05.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:05.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:05.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:05.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0adc20 2021-03-09T10:21:05.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d7c0 con 0x7ff99c0613a0 2021-03-09T10:21:05.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043ac0 con 0x7ff980050140 2021-03-09T10:21:05.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:05.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:05.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:05.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:05.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.063+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:05.073 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:05.063+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18092 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:05.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:05.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.067+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:05.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:05.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.067+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:05.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045f50 con 0x7ff980050140 2021-03-09T10:21:05.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980038490 con 0x7ff980050140 2021-03-09T10:21:05.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.067+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091ec0 secure :-1 s=READY pgs=5420 cs=0 l=1 rev1=1 rx=0x7ff980033930 tx=0x7ff994081720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:05.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.071+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940af000 con 0x7ff980050140 2021-03-09T10:21:05.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.075+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940369c0 con 0x7ff980050140 2021-03-09T10:21:05.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.075+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af8a0 con 0x7ff980050140 2021-03-09T10:21:05.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.075+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:05.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.075+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:05.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:05.075+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:10.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:21:10.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:10.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091ec0 secure :-1 s=READY pgs=5420 cs=0 l=1 rev1=1 rx=0x7ff980033930 tx=0x7ff994081720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:10.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091ec0 secure :-1 s=READY pgs=5420 cs=0 l=1 rev1=1 rx=0x7ff980033930 tx=0x7ff994081720).stop 2021-03-09T10:21:10.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:10.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091ec0 unknown :-1 s=CLOSED pgs=5420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:10.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:10.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:10.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:10.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0adc20 2021-03-09T10:21:10.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d7c0 con 0x7ff99c0613a0 2021-03-09T10:21:10.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000e130 con 0x7ff980051540 2021-03-09T10:21:10.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:10.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:10.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:10.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:10.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:10.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:10.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:10.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:10.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:10.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002e760 con 0x7ff980051540 2021-03-09T10:21:10.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045f50 con 0x7ff980051540 2021-03-09T10:21:10.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.067+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5428 cs=0 l=1 rev1=1 rx=0x7ff994063010 tx=0x7ff994095350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:10.078 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:10.067+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18131 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:10.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.071+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:21:10.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.071+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940af000 con 0x7ff980051540 2021-03-09T10:21:10.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.075+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940af8a0 con 0x7ff980051540 2021-03-09T10:21:10.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.075+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:10.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.075+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:10.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:10.075+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:15.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:15.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:15.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5428 cs=0 l=1 rev1=1 rx=0x7ff994063010 tx=0x7ff994095350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:15.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5428 cs=0 l=1 rev1=1 rx=0x7ff994063010 tx=0x7ff994095350).stop 2021-03-09T10:21:15.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:15.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:15.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:15.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:15.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:15.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0adc20 2021-03-09T10:21:15.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d7c0 con 0x7ff99c0613a0 2021-03-09T10:21:15.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:15.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037b90 con 0x7ff980050140 2021-03-09T10:21:15.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:15.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:15.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:15.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:15.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:15.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:15.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:15.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:15.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045f50 con 0x7ff980050140 2021-03-09T10:21:15.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002e760 con 0x7ff980050140 2021-03-09T10:21:15.086 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:15.071+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18166 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:15.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.071+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 secure :-1 s=READY pgs=5439 cs=0 l=1 rev1=1 rx=0x7ff9940952e0 tx=0x7ff99400bf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:15.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.075+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980050140 2021-03-09T10:21:15.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.075+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:21:15.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.079+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980050140 2021-03-09T10:21:15.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.079+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:15.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.079+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:15.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:15.079+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:20.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:20.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:20.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 secure :-1 s=READY pgs=5439 cs=0 l=1 rev1=1 rx=0x7ff9940952e0 tx=0x7ff99400bf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:20.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 secure :-1 s=READY pgs=5439 cs=0 l=1 rev1=1 rx=0x7ff9940952e0 tx=0x7ff99400bf40).stop 2021-03-09T10:21:20.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:20.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 unknown :-1 s=CLOSED pgs=5439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:20.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:20.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:20.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:20.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:20.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:20.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b280 con 0x7ff99c0adc20 2021-03-09T10:21:20.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d7c0 con 0x7ff99c0613a0 2021-03-09T10:21:20.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029ab0 con 0x7ff980051540 2021-03-09T10:21:20.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:20.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:20.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:20.084 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:20.075+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18205 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:20.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:20.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:20.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:20.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:20.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980038770 con 0x7ff980051540 2021-03-09T10:21:20.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.075+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000f000 con 0x7ff980051540 2021-03-09T10:21:20.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.079+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5451 cs=0 l=1 rev1=1 rx=0x7ff980003cb0 tx=0x7ff98c07d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:20.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.079+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980051540 2021-03-09T10:21:20.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.079+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07d0f0 con 0x7ff980051540 2021-03-09T10:21:20.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c07d0f0 con 0x7ff980051540 2021-03-09T10:21:20.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.087+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:20.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.087+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:20.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:20.087+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:25.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:25.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:25.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5451 cs=0 l=1 rev1=1 rx=0x7ff980003cb0 tx=0x7ff98c07d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:25.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5451 cs=0 l=1 rev1=1 rx=0x7ff980003cb0 tx=0x7ff98c07d460).stop 2021-03-09T10:21:25.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:25.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:25.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:25.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:25.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:25.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d7c0 con 0x7ff99c0adc20 2021-03-09T10:21:25.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043ac0 con 0x7ff99c0613a0 2021-03-09T10:21:25.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a7a0 con 0x7ff980050140 2021-03-09T10:21:25.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:25.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:25.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:25.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:25.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.079+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:25.087 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:25.079+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18240 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:25.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.083+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:25.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.083+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:25.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.083+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:25.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.083+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:25.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.083+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800148a0 con 0x7ff980050140 2021-03-09T10:21:25.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.083+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045f50 con 0x7ff980050140 2021-03-09T10:21:25.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.083+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 secure :-1 s=READY pgs=5466 cs=0 l=1 rev1=1 rx=0x7ff994095510 tx=0x7ff994014120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:25.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.083+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095120 con 0x7ff980050140 2021-03-09T10:21:25.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095120 con 0x7ff980050140 2021-03-09T10:21:25.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:21:25.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.091+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:25.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.091+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:25.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:25.091+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:30.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:21:30.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:30.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 secure :-1 s=READY pgs=5466 cs=0 l=1 rev1=1 rx=0x7ff994095510 tx=0x7ff994014120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:30.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 secure :-1 s=READY pgs=5466 cs=0 l=1 rev1=1 rx=0x7ff994095510 tx=0x7ff994014120).stop 2021-03-09T10:21:30.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980091e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:30.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980091e20 unknown :-1 s=CLOSED pgs=5466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:30.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:30.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:30.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:30.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010eb0 con 0x7ff99c0adc20 2021-03-09T10:21:30.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043ac0 con 0x7ff99c0613a0 2021-03-09T10:21:30.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035ba0 con 0x7ff980051540 2021-03-09T10:21:30.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:30.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:30.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:30.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:30.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.083+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:30.091 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:30.083+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18279 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:30.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.087+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:30.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.087+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:30.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.087+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:30.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.087+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:30.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.087+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000f1a0 con 0x7ff980051540 2021-03-09T10:21:30.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.087+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800148a0 con 0x7ff980051540 2021-03-09T10:21:30.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.087+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5475 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940a3300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:30.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081410 con 0x7ff980051540 2021-03-09T10:21:30.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081410 con 0x7ff980051540 2021-03-09T10:21:30.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:21:30.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.095+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:30.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.095+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:30.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:30.095+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:31.178 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:21:31.181 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:21:35.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:35.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:35.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5475 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940a3300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:35.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=5475 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940a3300).stop 2021-03-09T10:21:35.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:35.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:35.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:35.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:35.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:35.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043ac0 con 0x7ff99c0adc20 2021-03-09T10:21:35.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010eb0 con 0x7ff99c0613a0 2021-03-09T10:21:35.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000e2d0 con 0x7ff980050140 2021-03-09T10:21:35.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:35.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:35.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:35.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:35.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:35.094 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:35.087+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18314 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:35.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:35.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:35.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:35.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:35.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000e410 con 0x7ff980050140 2021-03-09T10:21:35.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.087+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980038770 con 0x7ff980050140 2021-03-09T10:21:35.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.091+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035b70 secure :-1 s=READY pgs=5483 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c07a070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:35.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:21:35.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a7950 con 0x7ff980050140 2021-03-09T10:21:35.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c07d0f0 con 0x7ff980050140 2021-03-09T10:21:35.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.099+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:35.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.099+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:35.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:35.099+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:40.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:40.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:40.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035b70 secure :-1 s=READY pgs=5483 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c07a070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:40.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035b70 secure :-1 s=READY pgs=5483 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c07a070).stop 2021-03-09T10:21:40.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:40.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035b70 unknown :-1 s=CLOSED pgs=5483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:40.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:40.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:40.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:40.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000bcf0 con 0x7ff99c0adc20 2021-03-09T10:21:40.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010eb0 con 0x7ff99c0613a0 2021-03-09T10:21:40.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043ac0 con 0x7ff980051540 2021-03-09T10:21:40.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:40.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:40.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:40.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:40.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.087+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:40.098 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:40.091+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18353 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:40.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:40.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:40.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:40.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:40.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff980051540 2021-03-09T10:21:40.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000f1a0 con 0x7ff980051540 2021-03-09T10:21:40.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5495 cs=0 l=1 rev1=1 rx=0x7ff98008c570 tx=0x7ff994095310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:40.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095120 con 0x7ff980051540 2021-03-09T10:21:40.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400add0 con 0x7ff980051540 2021-03-09T10:21:40.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:21:40.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.099+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:40.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.099+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:40.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:40.099+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:45.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:45.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:45.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5495 cs=0 l=1 rev1=1 rx=0x7ff98008c570 tx=0x7ff994095310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:45.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5495 cs=0 l=1 rev1=1 rx=0x7ff98008c570 tx=0x7ff994095310).stop 2021-03-09T10:21:45.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:45.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=CLOSED pgs=5495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:45.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:45.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:45.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980026120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:45.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043ae0 con 0x7ff99c0adc20 2021-03-09T10:21:45.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800167a0 con 0x7ff99c0613a0 2021-03-09T10:21:45.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800168e0 con 0x7ff980050140 2021-03-09T10:21:45.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:45.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:45.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:45.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:45.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.091+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980026120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:45.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:45.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:45.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:45.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:45.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093af0 con 0x7ff980050140 2021-03-09T10:21:45.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980025f90 con 0x7ff980050140 2021-03-09T10:21:45.103 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:45.095+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18388 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:45.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980026120 secure :-1 s=READY pgs=5503 cs=0 l=1 rev1=1 rx=0x7ff994063010 tx=0x7ff994006be0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:45.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff980050140 2021-03-09T10:21:45.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980050140 2021-03-09T10:21:45.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980050140 2021-03-09T10:21:45.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.103+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:45.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.103+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:45.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:45.103+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:50.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980026120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:50.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980026120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:50.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980026120 secure :-1 s=READY pgs=5503 cs=0 l=1 rev1=1 rx=0x7ff994063010 tx=0x7ff994006be0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:50.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980026120 secure :-1 s=READY pgs=5503 cs=0 l=1 rev1=1 rx=0x7ff994063010 tx=0x7ff994006be0).stop 2021-03-09T10:21:50.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980026120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:50.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980026120 unknown :-1 s=CLOSED pgs=5503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:50.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:50.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:50.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:50.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043ae0 con 0x7ff99c0adc20 2021-03-09T10:21:50.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093e20 con 0x7ff99c0613a0 2021-03-09T10:21:50.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff980051540 2021-03-09T10:21:50.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:50.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:50.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:50.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:50.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.095+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:50.105 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:50.099+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18427 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:50.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:50.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:50.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:50.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:50.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035ad0 con 0x7ff980051540 2021-03-09T10:21:50.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093af0 con 0x7ff980051540 2021-03-09T10:21:50.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5516 cs=0 l=1 rev1=1 rx=0x7ff99400af40 tx=0x7ff994081750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:50.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980051540 2021-03-09T10:21:50.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980051540 2021-03-09T10:21:50.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:21:50.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.107+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:50.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.107+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:50.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:50.107+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:21:55.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:55.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:55.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5516 cs=0 l=1 rev1=1 rx=0x7ff99400af40 tx=0x7ff994081750).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:55.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5516 cs=0 l=1 rev1=1 rx=0x7ff99400af40 tx=0x7ff994081750).stop 2021-03-09T10:21:55.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:55.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=CLOSED pgs=5516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:55.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:55.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:55.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:55.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043ae0 con 0x7ff99c0adc20 2021-03-09T10:21:55.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004acd0 con 0x7ff99c0613a0 2021-03-09T10:21:55.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026480 con 0x7ff980050140 2021-03-09T10:21:55.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:21:55.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:55.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:21:55.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:55.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:55.108 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:21:55.099+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18462 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:21:55.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:55.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:55.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:55.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:55.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035c60 con 0x7ff980050140 2021-03-09T10:21:55.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.099+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035ad0 con 0x7ff980050140 2021-03-09T10:21:55.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980025f90 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 rx=0x7ff9940b1560 tx=0x7ff994009d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:55.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940a30b0 con 0x7ff980050140 2021-03-09T10:21:55.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:21:55.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:21:55.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.111+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:55.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:55.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:21:55.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:00.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:00.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:00.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980025f90 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 rx=0x7ff9940b1560 tx=0x7ff994009d80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:00.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980025f90 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 rx=0x7ff9940b1560 tx=0x7ff994009d80).stop 2021-03-09T10:22:00.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:00.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980025f90 unknown :-1 s=CLOSED pgs=5526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:00.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:00.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:00.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:00.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027850 con 0x7ff99c0adc20 2021-03-09T10:22:00.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035f90 con 0x7ff99c0613a0 2021-03-09T10:22:00.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035ad0 con 0x7ff980051540 2021-03-09T10:22:00.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:00.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:00.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:00.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:00.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:00.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:00.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:00.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:00.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:00.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008e2c0 con 0x7ff980051540 2021-03-09T10:22:00.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035c60 con 0x7ff980051540 2021-03-09T10:22:00.113 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:00.103+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18501 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:00.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.103+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5535 cs=0 l=1 rev1=1 rx=0x7ff99400af90 tx=0x7ff99408cd10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:00.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980051540 2021-03-09T10:22:00.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99406f070 con 0x7ff980051540 2021-03-09T10:22:00.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:22:00.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.111+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:00.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:00.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:00.111+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:01.233 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:22:01.235 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:22:05.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:05.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:05.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5535 cs=0 l=1 rev1=1 rx=0x7ff99400af90 tx=0x7ff99408cd10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:05.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5535 cs=0 l=1 rev1=1 rx=0x7ff99400af90 tx=0x7ff99408cd10).stop 2021-03-09T10:22:05.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:05.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=CLOSED pgs=5535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:05.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:05.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:05.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:05.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a920 con 0x7ff99c0adc20 2021-03-09T10:22:05.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff99c0613a0 2021-03-09T10:22:05.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800260d0 con 0x7ff980050140 2021-03-09T10:22:05.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:05.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:05.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:05.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:05.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:05.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:05.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:05.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:05.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:05.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026240 con 0x7ff980050140 2021-03-09T10:22:05.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008e2c0 con 0x7ff980050140 2021-03-09T10:22:05.116 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:05.107+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18536 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:05.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5546 cs=0 l=1 rev1=1 rx=0x7ff980088560 tx=0x7ff994094b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:05.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.115+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980050140 2021-03-09T10:22:05.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.119+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980050140 2021-03-09T10:22:05.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.119+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:22:05.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.119+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:05.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.119+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:05.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:05.119+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:10.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:22:10.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.107+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:10.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5546 cs=0 l=1 rev1=1 rx=0x7ff980088560 tx=0x7ff994094b00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:10.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.107+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5546 cs=0 l=1 rev1=1 rx=0x7ff980088560 tx=0x7ff994094b00).stop 2021-03-09T10:22:10.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.107+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:10.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:10.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.107+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:10.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:10.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:10.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a9c0 con 0x7ff99c0adc20 2021-03-09T10:22:10.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff99c0613a0 2021-03-09T10:22:10.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:10.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800260d0 con 0x7ff980051540 2021-03-09T10:22:10.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:10.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:10.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:10.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:10.119 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:10.111+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18575 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:10.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:10.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:10.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:10.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:10.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008e270 con 0x7ff980051540 2021-03-09T10:22:10.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.111+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000e410 con 0x7ff980051540 2021-03-09T10:22:10.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.115+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5559 cs=0 l=1 rev1=1 rx=0x7ff980012290 tx=0x7ff98c00bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:10.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.115+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075080 con 0x7ff980051540 2021-03-09T10:22:10.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.123+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:22:10.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.123+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980051540 2021-03-09T10:22:10.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.123+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:10.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.123+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:10.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:10.123+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:15.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:15.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:15.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5559 cs=0 l=1 rev1=1 rx=0x7ff980012290 tx=0x7ff98c00bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:15.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 secure :-1 s=READY pgs=5559 cs=0 l=1 rev1=1 rx=0x7ff980012290 tx=0x7ff98c00bfd0).stop 2021-03-09T10:22:15.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980091e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:15.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980091e20 unknown :-1 s=CLOSED pgs=5559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:15.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:15.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:15.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:15.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000e540 con 0x7ff99c0adc20 2021-03-09T10:22:15.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a9c0 con 0x7ff99c0613a0 2021-03-09T10:22:15.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003ab00 con 0x7ff980050140 2021-03-09T10:22:15.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:15.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:15.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:15.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:15.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:15.123 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:15.115+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18610 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:15.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:15.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:15.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:15.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:15.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980038730 con 0x7ff980050140 2021-03-09T10:22:15.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.115+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026240 con 0x7ff980050140 2021-03-09T10:22:15.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.119+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5572 cs=0 l=1 rev1=1 rx=0x7ff9800448f0 tx=0x7ff994011400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:15.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.119+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99401fdf0 con 0x7ff980050140 2021-03-09T10:22:15.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.127+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400bed0 con 0x7ff980050140 2021-03-09T10:22:15.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.127+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:22:15.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.127+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:15.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.127+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:15.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:15.127+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:20.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:20.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:20.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5572 cs=0 l=1 rev1=1 rx=0x7ff9800448f0 tx=0x7ff994011400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:20.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5572 cs=0 l=1 rev1=1 rx=0x7ff9800448f0 tx=0x7ff994011400).stop 2021-03-09T10:22:20.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:20.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:20.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:20.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:20.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:20.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a670 con 0x7ff99c0adc20 2021-03-09T10:22:20.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800263e0 con 0x7ff99c0613a0 2021-03-09T10:22:20.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff980051540 2021-03-09T10:22:20.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:20.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:20.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:20.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:20.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:20.127 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:20.118+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18649 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:20.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:20.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:20.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:20.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:20.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800388c0 con 0x7ff980051540 2021-03-09T10:22:20.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008e270 con 0x7ff980051540 2021-03-09T10:22:20.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.118+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5578 cs=0 l=1 rev1=1 rx=0x7ff98c0a7ac0 tx=0x7ff98c0a9ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:20.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075050 con 0x7ff980051540 2021-03-09T10:22:20.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a7720 con 0x7ff980051540 2021-03-09T10:22:20.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:22:20.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.126+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:20.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.126+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:20.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:20.126+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:25.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:22:25.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:25.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5578 cs=0 l=1 rev1=1 rx=0x7ff98c0a7ac0 tx=0x7ff98c0a9ff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:25.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5578 cs=0 l=1 rev1=1 rx=0x7ff98c0a7ac0 tx=0x7ff98c0a9ff0).stop 2021-03-09T10:22:25.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:25.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=CLOSED pgs=5578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:25.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:25.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:25.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:25.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014500 con 0x7ff99c0adc20 2021-03-09T10:22:25.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:25.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a670 con 0x7ff99c0613a0 2021-03-09T10:22:25.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800537c0 con 0x7ff980050140 2021-03-09T10:22:25.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.118+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:25.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.122+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:25.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.122+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:25.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:25.143 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:25.122+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18684 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:25.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:25.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:25.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:25.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:25.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800263e0 con 0x7ff980050140 2021-03-09T10:22:25.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980038730 con 0x7ff980050140 2021-03-09T10:22:25.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5592 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff99400efd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:25.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994001370 con 0x7ff980050140 2021-03-09T10:22:25.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.126+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994001370 con 0x7ff980050140 2021-03-09T10:22:25.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:22:25.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.134+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:25.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.134+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:25.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:25.134+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:30.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:30.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:30.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5592 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff99400efd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:30.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5592 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff99400efd0).stop 2021-03-09T10:22:30.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:30.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:30.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:30.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:30.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:30.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a670 con 0x7ff99c0adc20 2021-03-09T10:22:30.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038730 con 0x7ff99c0613a0 2021-03-09T10:22:30.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff980051540 2021-03-09T10:22:30.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:30.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:30.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.122+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:30.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:30.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:30.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.126+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:30.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:30.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.126+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:30.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.126+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:30.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.126+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008e410 con 0x7ff980051540 2021-03-09T10:22:30.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.126+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800263e0 con 0x7ff980051540 2021-03-09T10:22:30.136 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:30.126+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18723 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:30.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5599 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99400c620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:30.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081410 con 0x7ff980051540 2021-03-09T10:22:30.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098d40 con 0x7ff980051540 2021-03-09T10:22:30.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:22:30.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.134+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:30.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.134+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:30.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:30.134+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:31.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:22:31.296 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:22:35.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:35.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:35.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5599 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99400c620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:35.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5599 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99400c620).stop 2021-03-09T10:22:35.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:35.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=CLOSED pgs=5599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:35.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:35.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:35.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:35.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003e1e0 con 0x7ff99c0adc20 2021-03-09T10:22:35.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026b40 con 0x7ff99c0613a0 2021-03-09T10:22:35.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800095c0 con 0x7ff980050140 2021-03-09T10:22:35.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:35.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:35.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:35.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:35.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:35.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:35.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:35.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:35.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:35.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980038730 con 0x7ff980050140 2021-03-09T10:22:35.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008e410 con 0x7ff980050140 2021-03-09T10:22:35.140 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:35.130+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18758 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:35.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.130+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5610 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940aa130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:35.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009a20 con 0x7ff980050140 2021-03-09T10:22:35.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.138+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009a20 con 0x7ff980050140 2021-03-09T10:22:35.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.138+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980050140 2021-03-09T10:22:35.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.138+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:35.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.138+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:35.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:35.138+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:40.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:40.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:40.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5610 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940aa130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:40.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5610 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940aa130).stop 2021-03-09T10:22:40.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:40.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:40.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:40.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:40.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:40.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:40.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e410 con 0x7ff99c0adc20 2021-03-09T10:22:40.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:40.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff99c0613a0 2021-03-09T10:22:40.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003aeb0 con 0x7ff980051540 2021-03-09T10:22:40.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:40.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:40.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:40.143 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:40.134+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18797 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:40.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:40.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:40.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:40.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:40.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800263e0 con 0x7ff980051540 2021-03-09T10:22:40.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.134+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980038730 con 0x7ff980051540 2021-03-09T10:22:40.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.138+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5625 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99400d6b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:40.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.138+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980051540 2021-03-09T10:22:40.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.142+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980051540 2021-03-09T10:22:40.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.142+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:22:40.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.142+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:40.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.142+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:40.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:40.142+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:45.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:45.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:45.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5625 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99400d6b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:45.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5625 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99400d6b0).stop 2021-03-09T10:22:45.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:45.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=CLOSED pgs=5625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:45.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:45.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:45.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:45.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038730 con 0x7ff99c0adc20 2021-03-09T10:22:45.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e410 con 0x7ff99c0613a0 2021-03-09T10:22:45.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088e50 con 0x7ff980050140 2021-03-09T10:22:45.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:45.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:45.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:45.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:45.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:45.146 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:45.138+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18831 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:45.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:45.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:45.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:45.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:45.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088f90 con 0x7ff980050140 2021-03-09T10:22:45.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.138+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800263e0 con 0x7ff980050140 2021-03-09T10:22:45.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.142+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5635 cs=0 l=1 rev1=1 rx=0x7ff99400d790 tx=0x7ff9940013b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:45.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.142+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99402a180 con 0x7ff980050140 2021-03-09T10:22:45.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.146+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994034b80 con 0x7ff980050140 2021-03-09T10:22:45.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.150+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:22:45.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.150+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:45.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.150+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:45.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:45.150+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:50.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:50.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:50.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5635 cs=0 l=1 rev1=1 rx=0x7ff99400d790 tx=0x7ff9940013b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:50.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5635 cs=0 l=1 rev1=1 rx=0x7ff99400d790 tx=0x7ff9940013b0).stop 2021-03-09T10:22:50.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:50.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:50.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:50.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:50.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:50.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094570 con 0x7ff99c0adc20 2021-03-09T10:22:50.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f470 con 0x7ff99c0613a0 2021-03-09T10:22:50.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980038730 con 0x7ff980051540 2021-03-09T10:22:50.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:50.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:50.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:50.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:50.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:50.150 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:50.142+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18871 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:50.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:50.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:50.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:50.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:50.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008e410 con 0x7ff980051540 2021-03-09T10:22:50.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800388c0 con 0x7ff980051540 2021-03-09T10:22:50.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.142+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5642 cs=0 l=1 rev1=1 rx=0x7ff98c074f70 tx=0x7ff98c0a7a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:50.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.146+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980051540 2021-03-09T10:22:50.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.146+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a7d60 con 0x7ff980051540 2021-03-09T10:22:50.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.150+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980051540 2021-03-09T10:22:50.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.150+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:50.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.150+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:50.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:50.150+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:22:55.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:55.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:55.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5642 cs=0 l=1 rev1=1 rx=0x7ff98c074f70 tx=0x7ff98c0a7a20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:55.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5642 cs=0 l=1 rev1=1 rx=0x7ff98c074f70 tx=0x7ff98c0a7a20).stop 2021-03-09T10:22:55.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:55.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=CLOSED pgs=5642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:55.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:55.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:55.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:55.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094520 con 0x7ff99c0adc20 2021-03-09T10:22:55.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ddd0 con 0x7ff99c0613a0 2021-03-09T10:22:55.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008df10 con 0x7ff980050140 2021-03-09T10:22:55.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:22:55.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:55.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:22:55.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:55.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:55.154 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:22:55.146+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18905 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:22:55.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:55.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:55.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:55.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:55.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800263e0 con 0x7ff980050140 2021-03-09T10:22:55.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.146+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088f90 con 0x7ff980050140 2021-03-09T10:22:55.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.150+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5658 cs=0 l=1 rev1=1 rx=0x7ff99407a270 tx=0x7ff994009d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:55.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.150+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994066650 con 0x7ff980050140 2021-03-09T10:22:55.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.154+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994041550 con 0x7ff980050140 2021-03-09T10:22:55.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.158+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:22:55.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.158+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:55.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.158+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:55.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:22:55.158+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:00.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:00.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:00.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5658 cs=0 l=1 rev1=1 rx=0x7ff99407a270 tx=0x7ff994009d80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:00.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5658 cs=0 l=1 rev1=1 rx=0x7ff99407a270 tx=0x7ff994009d80).stop 2021-03-09T10:23:00.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:00.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:00.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:00.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:00.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:00.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088b10 con 0x7ff99c0adc20 2021-03-09T10:23:00.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057350 con 0x7ff99c0613a0 2021-03-09T10:23:00.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff980051540 2021-03-09T10:23:00.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:00.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:00.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:00.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:00.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:00.157 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:00.150+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18945 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:00.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:00.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:00.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:00.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:00.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088e00 con 0x7ff980051540 2021-03-09T10:23:00.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008e410 con 0x7ff980051540 2021-03-09T10:23:00.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5664 cs=0 l=1 rev1=1 rx=0x7ff98c0172e0 tx=0x7ff98c07e280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:00.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075040 con 0x7ff980051540 2021-03-09T10:23:00.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.150+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980051540 2021-03-09T10:23:00.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.158+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980051540 2021-03-09T10:23:00.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.158+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:00.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.158+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:00.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:00.158+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:01.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:23:01.362 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:23:05.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:05.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:05.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5664 cs=0 l=1 rev1=1 rx=0x7ff98c0172e0 tx=0x7ff98c07e280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:05.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5664 cs=0 l=1 rev1=1 rx=0x7ff98c0172e0 tx=0x7ff98c07e280).stop 2021-03-09T10:23:05.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:05.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=CLOSED pgs=5664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:05.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:05.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:05.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:05.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057330 con 0x7ff99c0adc20 2021-03-09T10:23:05.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e410 con 0x7ff99c0613a0 2021-03-09T10:23:05.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088b10 con 0x7ff980050140 2021-03-09T10:23:05.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:05.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:05.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:05.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:05.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:05.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:05.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:05.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:05.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:05.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008a9e0 con 0x7ff980050140 2021-03-09T10:23:05.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800263e0 con 0x7ff980050140 2021-03-09T10:23:05.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5673 cs=0 l=1 rev1=1 rx=0x7ff9940858d0 tx=0x7ff994066650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:05.167 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:05.154+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18979 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:05.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980050140 2021-03-09T10:23:05.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.154+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980050140 2021-03-09T10:23:05.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.158+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980050140 2021-03-09T10:23:05.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.158+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:05.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.158+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:05.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:05.158+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:10.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:23:10.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:10.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5673 cs=0 l=1 rev1=1 rx=0x7ff9940858d0 tx=0x7ff994066650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:10.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5673 cs=0 l=1 rev1=1 rx=0x7ff9940858d0 tx=0x7ff994066650).stop 2021-03-09T10:23:10.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:10.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:10.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:10.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:10.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:10.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043870 con 0x7ff99c0adc20 2021-03-09T10:23:10.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004db30 con 0x7ff99c0613a0 2021-03-09T10:23:10.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e410 con 0x7ff980051540 2021-03-09T10:23:10.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:10.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:10.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:10.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:10.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:10.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:10.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:10.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:10.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:10.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800263e0 con 0x7ff980051540 2021-03-09T10:23:10.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008a9e0 con 0x7ff980051540 2021-03-09T10:23:10.168 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:10.158+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19019 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:10.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.158+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5685 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994081750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:10.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.162+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994001210 con 0x7ff980051540 2021-03-09T10:23:10.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.162+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994001210 con 0x7ff980051540 2021-03-09T10:23:10.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069ea0 con 0x7ff980051540 2021-03-09T10:23:10.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.166+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:10.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.166+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:10.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:10.166+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:15.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:23:15.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:15.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5685 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994081750).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:15.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5685 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994081750).stop 2021-03-09T10:23:15.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:15.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=CLOSED pgs=5685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:15.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:15.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:15.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:15.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004db30 con 0x7ff99c0adc20 2021-03-09T10:23:15.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e410 con 0x7ff99c0613a0 2021-03-09T10:23:15.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800498f0 con 0x7ff980050140 2021-03-09T10:23:15.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:15.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:15.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:15.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:15.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:15.169 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:15.162+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19053 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:15.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:15.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:15.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:15.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:15.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980057310 con 0x7ff980050140 2021-03-09T10:23:15.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088e00 con 0x7ff980050140 2021-03-09T10:23:15.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.162+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5696 cs=0 l=1 rev1=1 rx=0x7ff98c084ad0 tx=0x7ff98c00a210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:15.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c075070 con 0x7ff980050140 2021-03-09T10:23:15.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.170+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980050140 2021-03-09T10:23:15.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.174+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980050140 2021-03-09T10:23:15.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.174+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:15.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.174+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:15.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:15.174+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:20.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:20.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:20.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5696 cs=0 l=1 rev1=1 rx=0x7ff98c084ad0 tx=0x7ff98c00a210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:20.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5696 cs=0 l=1 rev1=1 rx=0x7ff98c084ad0 tx=0x7ff98c00a210).stop 2021-03-09T10:23:20.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:20.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:20.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:20.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:20.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:20.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004db30 con 0x7ff99c0adc20 2021-03-09T10:23:20.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e410 con 0x7ff99c0613a0 2021-03-09T10:23:20.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003aeb0 con 0x7ff980051540 2021-03-09T10:23:20.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:20.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:20.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:20.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:20.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:20.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:20.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:20.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:20.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:20.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800498a0 con 0x7ff980051540 2021-03-09T10:23:20.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980057310 con 0x7ff980051540 2021-03-09T10:23:20.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5705 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c022210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:20.176 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:20.166+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19093 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:20.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08fed0 con 0x7ff980051540 2021-03-09T10:23:20.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:23:20.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.170+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:23:20.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.170+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:20.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.170+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:20.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:20.170+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:25.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:25.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:25.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5705 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c022210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:25.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5705 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c022210).stop 2021-03-09T10:23:25.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:25.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=CLOSED pgs=5705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:25.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:25.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:25.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:25.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004db30 con 0x7ff99c0adc20 2021-03-09T10:23:25.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049a30 con 0x7ff99c0613a0 2021-03-09T10:23:25.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049b70 con 0x7ff980050140 2021-03-09T10:23:25.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:25.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:25.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:25.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:25.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:25.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:25.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:25.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:25.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:25.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008e1b0 con 0x7ff980050140 2021-03-09T10:23:25.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.166+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800263e0 con 0x7ff980050140 2021-03-09T10:23:25.179 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:25.170+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19127 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:25.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.170+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5718 cs=0 l=1 rev1=1 rx=0x7ff994065040 tx=0x7ff994095f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:25.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.178+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994006bf0 con 0x7ff980050140 2021-03-09T10:23:25.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.178+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994068730 con 0x7ff980050140 2021-03-09T10:23:25.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.178+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:23:25.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.178+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:25.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.178+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:25.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:25.178+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:30.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:30.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:30.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5718 cs=0 l=1 rev1=1 rx=0x7ff994065040 tx=0x7ff994095f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:30.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5718 cs=0 l=1 rev1=1 rx=0x7ff994065040 tx=0x7ff994095f00).stop 2021-03-09T10:23:30.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:30.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:30.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:30.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:30.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:30.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093d90 con 0x7ff99c0adc20 2021-03-09T10:23:30.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045d60 con 0x7ff99c0613a0 2021-03-09T10:23:30.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004db30 con 0x7ff980051540 2021-03-09T10:23:30.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:30.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:30.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:30.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:30.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.170+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:30.182 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:30.174+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19167 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:30.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.174+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:30.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.174+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:30.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.174+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:30.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.174+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:30.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.174+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800263e0 con 0x7ff980051540 2021-03-09T10:23:30.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.174+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008e1b0 con 0x7ff980051540 2021-03-09T10:23:30.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.174+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5730 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99407a2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:30.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.178+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:23:30.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.178+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:23:30.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.182+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:23:30.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.182+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:30.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.182+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:30.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:30.182+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:31.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:23:31.417 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:23:35.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:35.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:35.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5730 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99407a2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:35.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 secure :-1 s=READY pgs=5730 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99407a2e0).stop 2021-03-09T10:23:35.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98003a9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:35.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98003a9c0 unknown :-1 s=CLOSED pgs=5730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:35.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:35.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:35.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:35.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004db30 con 0x7ff99c0adc20 2021-03-09T10:23:35.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094d80 con 0x7ff99c0613a0 2021-03-09T10:23:35.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e350 con 0x7ff980050140 2021-03-09T10:23:35.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:35.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:35.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:35.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:35.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:35.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:35.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:35.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:35.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.173+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:35.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.177+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088e00 con 0x7ff980050140 2021-03-09T10:23:35.186 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:35.177+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19201 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:35.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.177+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800263e0 con 0x7ff980050140 2021-03-09T10:23:35.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.177+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5740 cs=0 l=1 rev1=1 rx=0x7ff994014510 tx=0x7ff9940861e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:35.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980050140 2021-03-09T10:23:35.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.185+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:23:35.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.185+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980050140 2021-03-09T10:23:35.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.185+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:35.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.185+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:35.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:35.185+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:40.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:40.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:40.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5740 cs=0 l=1 rev1=1 rx=0x7ff994014510 tx=0x7ff9940861e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:40.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5740 cs=0 l=1 rev1=1 rx=0x7ff994014510 tx=0x7ff9940861e0).stop 2021-03-09T10:23:40.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:40.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:40.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:40.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:40.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:40.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e350 con 0x7ff99c0adc20 2021-03-09T10:23:40.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800263e0 con 0x7ff99c0613a0 2021-03-09T10:23:40.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a9c0 con 0x7ff980051540 2021-03-09T10:23:40.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:40.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:40.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:40.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:40.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:40.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:40.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:40.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:40.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:40.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003ab00 con 0x7ff980051540 2021-03-09T10:23:40.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800498a0 con 0x7ff980051540 2021-03-09T10:23:40.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 secure :-1 s=READY pgs=5750 cs=0 l=1 rev1=1 rx=0x7ff98c0a1010 tx=0x7ff98c00ba80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:40.194 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:40.181+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19241 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:40.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08fed0 con 0x7ff980051540 2021-03-09T10:23:40.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff980051540 2021-03-09T10:23:40.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.189+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:23:40.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.189+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:40.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.189+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:40.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:40.189+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:45.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980049a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:23:45.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980049a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:45.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 secure :-1 s=READY pgs=5750 cs=0 l=1 rev1=1 rx=0x7ff98c0a1010 tx=0x7ff98c00ba80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:45.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 secure :-1 s=READY pgs=5750 cs=0 l=1 rev1=1 rx=0x7ff98c0a1010 tx=0x7ff98c00ba80).stop 2021-03-09T10:23:45.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980049a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:45.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 unknown :-1 s=CLOSED pgs=5750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:45.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:45.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:45.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:45.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e1b0 con 0x7ff99c0adc20 2021-03-09T10:23:45.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e2f0 con 0x7ff99c0613a0 2021-03-09T10:23:45.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026210 con 0x7ff980050140 2021-03-09T10:23:45.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:45.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:45.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:45.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:45.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:45.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:45.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:45.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:45.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:45.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026350 con 0x7ff980050140 2021-03-09T10:23:45.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.181+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088e00 con 0x7ff980050140 2021-03-09T10:23:45.194 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:45.181+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19275 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:45.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.185+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99407a1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:45.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.189+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994001d00 con 0x7ff980050140 2021-03-09T10:23:45.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.193+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994001d00 con 0x7ff980050140 2021-03-09T10:23:45.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.193+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:23:45.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.193+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:45.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.193+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:45.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:45.193+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:49.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 msgr2=0x7ff980039e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:23:49.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 msgr2=0x7ff980039e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:49.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff980039e80 secure :-1 s=READY pgs=3675 cs=0 l=1 rev1=1 rx=0x7ff984003940 tx=0x7ff984000d00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:49.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff980039e80 secure :-1 s=READY pgs=3675 cs=0 l=1 rev1=1 rx=0x7ff984003940 tx=0x7ff984000d00).stop 2021-03-09T10:23:49.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 msgr2=0x7ff980039e80 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:23:49.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff9937fe700 4 mgrc ms_handle_reset ms_handle_reset con 0x7ff980083100 2021-03-09T10:23:49.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff9937fe700 4 mgrc reconnect Terminating session with v2:172.21.15.52:6808/14433 2021-03-09T10:23:49.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 msgr2=0x7ff980039e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:49.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff980039e80 unknown :-1 s=CLOSED pgs=3675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:49.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff9937fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:49.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 0x7ff980049a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:49.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 0x7ff980049a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:49.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:49.897+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 0x7ff980049a30 secure :-1 s=READY pgs=5484 cs=0 l=1 rev1=1 rx=0x7ff984049060 tx=0x7ff984000d00).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:50.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:23:50.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:50.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99407a1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:50.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99407a1f0).stop 2021-03-09T10:23:50.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:50.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:50.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:50.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:50.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:50.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026210 con 0x7ff99c0adc20 2021-03-09T10:23:50.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800363f0 con 0x7ff99c0613a0 2021-03-09T10:23:50.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f270 con 0x7ff980083100 2021-03-09T10:23:50.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:50.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:50.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:50.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:50.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:50.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:50.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:50.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:50.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:50.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088e00 con 0x7ff980083100 2021-03-09T10:23:50.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.185+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026350 con 0x7ff980083100 2021-03-09T10:23:50.201 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:50.185+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19315 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:50.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.189+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e80 secure :-1 s=READY pgs=5769 cs=0 l=1 rev1=1 rx=0x7ff980093fe0 tx=0x7ff99409b270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:50.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.189+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994006bf0 con 0x7ff980083100 2021-03-09T10:23:50.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.193+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994006bf0 con 0x7ff980083100 2021-03-09T10:23:50.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:23:50.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.197+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:50.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.197+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:50.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:50.197+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:23:55.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:23:55.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:55.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e80 secure :-1 s=READY pgs=5769 cs=0 l=1 rev1=1 rx=0x7ff980093fe0 tx=0x7ff99409b270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:55.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e80 secure :-1 s=READY pgs=5769 cs=0 l=1 rev1=1 rx=0x7ff980093fe0 tx=0x7ff99409b270).stop 2021-03-09T10:23:55.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:55.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e80 unknown :-1 s=CLOSED pgs=5769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:55.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:55.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:55.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:55.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026210 con 0x7ff99c0adc20 2021-03-09T10:23:55.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026350 con 0x7ff99c0613a0 2021-03-09T10:23:55.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.189+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f220 con 0x7ff980050140 2021-03-09T10:23:55.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.193+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:23:55.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.193+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:55.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.193+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:23:55.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.193+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:55.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.193+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:55.199 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:23:55.193+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19349 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:23:55.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.197+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:55.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.197+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:55.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.197+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:55.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.197+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:55.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.197+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004f360 con 0x7ff980050140 2021-03-09T10:23:55.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.197+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088e00 con 0x7ff980050140 2021-03-09T10:23:55.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.197+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5786 cs=0 l=1 rev1=1 rx=0x7ff980044810 tx=0x7ff994006bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:55.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009a20 con 0x7ff980050140 2021-03-09T10:23:55.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.201+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009a20 con 0x7ff980050140 2021-03-09T10:23:55.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.205+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99400d380 con 0x7ff980050140 2021-03-09T10:23:55.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.205+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:55.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.205+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:55.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:23:55.205+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:00.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.193+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:00.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.193+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:00.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.193+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5786 cs=0 l=1 rev1=1 rx=0x7ff980044810 tx=0x7ff994006bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:00.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.193+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 secure :-1 s=READY pgs=5786 cs=0 l=1 rev1=1 rx=0x7ff980044810 tx=0x7ff994006bf0).stop 2021-03-09T10:24:00.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.193+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:00.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.193+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035ad0 unknown :-1 s=CLOSED pgs=5786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:00.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:00.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:00.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:00.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:00.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:00.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f1d0 con 0x7ff99c0adc20 2021-03-09T10:24:00.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a320 con 0x7ff99c0613a0 2021-03-09T10:24:00.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800531c0 con 0x7ff980083100 2021-03-09T10:24:00.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:00.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:00.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:00.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:00.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:00.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:00.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:00.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980053330 con 0x7ff980083100 2021-03-09T10:24:00.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003ab00 con 0x7ff980083100 2021-03-09T10:24:00.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e30 secure :-1 s=READY pgs=5791 cs=0 l=1 rev1=1 rx=0x7ff98c00b5d0 tx=0x7ff98c084150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:00.206 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:00.197+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19389 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:00.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:24:00.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079fc0 con 0x7ff980083100 2021-03-09T10:24:00.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.205+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:24:00.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.205+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:00.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.205+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:00.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:00.205+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:01.460 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:24:01.463 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:24:05.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:05.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:05.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e30 secure :-1 s=READY pgs=5791 cs=0 l=1 rev1=1 rx=0x7ff98c00b5d0 tx=0x7ff98c084150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:05.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e30 secure :-1 s=READY pgs=5791 cs=0 l=1 rev1=1 rx=0x7ff98c00b5d0 tx=0x7ff98c084150).stop 2021-03-09T10:24:05.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:05.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039e30 unknown :-1 s=CLOSED pgs=5791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:05.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:05.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:05.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:05.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800534c0 con 0x7ff99c0adc20 2021-03-09T10:24:05.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c0d0 con 0x7ff99c0613a0 2021-03-09T10:24:05.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045b90 con 0x7ff980050140 2021-03-09T10:24:05.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:05.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:05.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:05.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.197+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:05.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.201+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:05.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.201+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:05.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.201+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:05.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.201+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:05.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.201+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:05.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.201+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004f1d0 con 0x7ff980050140 2021-03-09T10:24:05.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.201+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980053330 con 0x7ff980050140 2021-03-09T10:24:05.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.201+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035aa0 secure :-1 s=READY pgs=5801 cs=0 l=1 rev1=1 rx=0x7ff98c08d230 tx=0x7ff98c099ce0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:05.209 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:05.201+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19423 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:05.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.205+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04bd70 con 0x7ff980050140 2021-03-09T10:24:05.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.205+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c072970 con 0x7ff980050140 2021-03-09T10:24:05.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.205+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072ae0 con 0x7ff980050140 2021-03-09T10:24:05.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.205+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:05.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.205+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:05.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:05.205+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:10.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:10.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:10.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035aa0 secure :-1 s=READY pgs=5801 cs=0 l=1 rev1=1 rx=0x7ff98c08d230 tx=0x7ff98c099ce0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:10.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035aa0 secure :-1 s=READY pgs=5801 cs=0 l=1 rev1=1 rx=0x7ff98c08d230 tx=0x7ff98c099ce0).stop 2021-03-09T10:24:10.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980035aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:10.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980035aa0 unknown :-1 s=CLOSED pgs=5801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:10.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:10.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:10.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:10.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039e30 con 0x7ff99c0adc20 2021-03-09T10:24:10.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039f70 con 0x7ff99c0613a0 2021-03-09T10:24:10.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a0b0 con 0x7ff980083100 2021-03-09T10:24:10.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:10.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:10.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:10.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.201+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:10.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.205+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:10.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.205+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:10.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.205+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:10.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.205+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:10.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.205+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:10.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.205+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003aca0 con 0x7ff980083100 2021-03-09T10:24:10.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.205+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004f360 con 0x7ff980083100 2021-03-09T10:24:10.213 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:10.205+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19463 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:10.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.205+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5814 cs=0 l=1 rev1=1 rx=0x7ff980038600 tx=0x7ff994011490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:10.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.209+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408dd40 con 0x7ff980083100 2021-03-09T10:24:10.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.209+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980083100 2021-03-09T10:24:10.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.213+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:24:10.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.213+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:10.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.213+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:10.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:10.213+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:15.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:24:15.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:15.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5814 cs=0 l=1 rev1=1 rx=0x7ff980038600 tx=0x7ff994011490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:15.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5814 cs=0 l=1 rev1=1 rx=0x7ff980038600 tx=0x7ff994011490).stop 2021-03-09T10:24:15.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:15.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=CLOSED pgs=5814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:15.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:15.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:15.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:15.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a320 con 0x7ff99c0adc20 2021-03-09T10:24:15.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035c80 con 0x7ff99c0613a0 2021-03-09T10:24:15.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff980050140 2021-03-09T10:24:15.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:15.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.205+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:15.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.209+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:15.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.209+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:15.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.209+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:15.214 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:15.209+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19497 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:15.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.209+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:15.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.209+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:15.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.209+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:15.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.209+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:15.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.209+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035f00 con 0x7ff980050140 2021-03-09T10:24:15.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.209+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003aca0 con 0x7ff980050140 2021-03-09T10:24:15.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.213+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5825 cs=0 l=1 rev1=1 rx=0x7ff980057250 tx=0x7ff9940aa320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:15.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.213+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408dd40 con 0x7ff980050140 2021-03-09T10:24:15.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.217+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408dd40 con 0x7ff980050140 2021-03-09T10:24:15.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.221+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:24:15.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.221+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:15.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.221+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:15.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:15.221+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:20.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:20.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:20.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5825 cs=0 l=1 rev1=1 rx=0x7ff980057250 tx=0x7ff9940aa320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:20.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5825 cs=0 l=1 rev1=1 rx=0x7ff980057250 tx=0x7ff9940aa320).stop 2021-03-09T10:24:20.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:20.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=CLOSED pgs=5825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:20.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:20.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:20.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:20.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035c80 con 0x7ff99c0adc20 2021-03-09T10:24:20.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035dc0 con 0x7ff99c0613a0 2021-03-09T10:24:20.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003ab00 con 0x7ff980083100 2021-03-09T10:24:20.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:20.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:20.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:20.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.209+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:20.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.213+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:20.218 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:20.213+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19537 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:20.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.213+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:20.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.213+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:20.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.213+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:20.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.213+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:20.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.213+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003ac40 con 0x7ff980083100 2021-03-09T10:24:20.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.213+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004f1d0 con 0x7ff980083100 2021-03-09T10:24:20.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.213+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5835 cs=0 l=1 rev1=1 rx=0x7ff9800361d0 tx=0x7ff98c04bf60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:20.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.217+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff980083100 2021-03-09T10:24:20.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.217+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00ba80 con 0x7ff980083100 2021-03-09T10:24:20.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.225+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c099d50 con 0x7ff980083100 2021-03-09T10:24:20.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.225+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:20.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.225+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:20.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:20.225+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:25.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:24:25.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:25.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5835 cs=0 l=1 rev1=1 rx=0x7ff9800361d0 tx=0x7ff98c04bf60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:25.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5835 cs=0 l=1 rev1=1 rx=0x7ff9800361d0 tx=0x7ff98c04bf60).stop 2021-03-09T10:24:25.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:25.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=CLOSED pgs=5835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:25.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:25.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:25.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:25.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800588d0 con 0x7ff99c0adc20 2021-03-09T10:24:25.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053600 con 0x7ff99c0613a0 2021-03-09T10:24:25.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003aac0 con 0x7ff980050140 2021-03-09T10:24:25.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:25.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:25.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:25.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:25.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.213+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:25.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.217+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:25.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.217+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:25.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.217+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:25.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.217+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:25.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.217+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a320 con 0x7ff980050140 2021-03-09T10:24:25.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.217+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035f00 con 0x7ff980050140 2021-03-09T10:24:25.224 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:25.217+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19571 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:25.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.217+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5846 cs=0 l=1 rev1=1 rx=0x7ff9940b2050 tx=0x7ff99406df80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:25.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.221+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff980050140 2021-03-09T10:24:25.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.221+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff980050140 2021-03-09T10:24:25.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.225+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:24:25.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.225+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:25.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.225+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:25.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:25.225+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:30.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:30.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:30.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5846 cs=0 l=1 rev1=1 rx=0x7ff9940b2050 tx=0x7ff99406df80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:30.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5846 cs=0 l=1 rev1=1 rx=0x7ff9940b2050 tx=0x7ff99406df80).stop 2021-03-09T10:24:30.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:30.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=CLOSED pgs=5846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:30.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:30.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:30.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:30.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800588d0 con 0x7ff99c0adc20 2021-03-09T10:24:30.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003aac0 con 0x7ff99c0613a0 2021-03-09T10:24:30.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053600 con 0x7ff980083100 2021-03-09T10:24:30.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:30.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:30.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:30.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:30.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.217+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:30.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.221+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:30.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.221+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:30.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.221+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:30.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.221+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:30.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.221+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a320 con 0x7ff980083100 2021-03-09T10:24:30.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.221+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003ac40 con 0x7ff980083100 2021-03-09T10:24:30.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.221+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5855 cs=0 l=1 rev1=1 rx=0x7ff98c0a02f0 tx=0x7ff98c00bef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:30.228 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:30.221+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19611 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:30.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.221+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04bd70 con 0x7ff980083100 2021-03-09T10:24:30.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.221+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a0130 con 0x7ff980083100 2021-03-09T10:24:30.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.225+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c016d40 con 0x7ff980083100 2021-03-09T10:24:30.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.229+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:30.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.229+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:30.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:30.229+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:31.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:24:31.578 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:24:35.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:35.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:35.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5855 cs=0 l=1 rev1=1 rx=0x7ff98c0a02f0 tx=0x7ff98c00bef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:35.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5855 cs=0 l=1 rev1=1 rx=0x7ff98c0a02f0 tx=0x7ff98c00bef0).stop 2021-03-09T10:24:35.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:35.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=CLOSED pgs=5855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:35.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:35.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:35.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:35.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003aac0 con 0x7ff99c0adc20 2021-03-09T10:24:35.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053600 con 0x7ff99c0613a0 2021-03-09T10:24:35.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800459b0 con 0x7ff980050140 2021-03-09T10:24:35.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:35.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:35.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:35.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.221+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:35.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.225+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:35.230 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:35.225+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19645 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:35.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.225+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:35.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.225+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:35.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.225+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:35.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.225+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:35.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.225+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a320 con 0x7ff980050140 2021-03-09T10:24:35.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.225+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003ac40 con 0x7ff980050140 2021-03-09T10:24:35.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.229+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5871 cs=0 l=1 rev1=1 rx=0x7ff98c098040 tx=0x7ff98c07d160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:35.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.229+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c07ee20 con 0x7ff980050140 2021-03-09T10:24:35.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980050140 2021-03-09T10:24:35.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.237+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980050140 2021-03-09T10:24:35.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.237+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:35.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.237+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:35.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:35.237+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:40.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:24:40.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:40.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5871 cs=0 l=1 rev1=1 rx=0x7ff98c098040 tx=0x7ff98c07d160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:40.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5871 cs=0 l=1 rev1=1 rx=0x7ff98c098040 tx=0x7ff98c07d160).stop 2021-03-09T10:24:40.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:40.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=CLOSED pgs=5871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:40.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:40.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:40.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:40.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001bbc0 con 0x7ff99c0adc20 2021-03-09T10:24:40.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003aac0 con 0x7ff99c0613a0 2021-03-09T10:24:40.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053600 con 0x7ff980083100 2021-03-09T10:24:40.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:40.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:40.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:40.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.225+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:40.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.229+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:40.238 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:40.229+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19684 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:40.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.233+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:40.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.233+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:40.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.233+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:40.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.233+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:40.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.233+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a320 con 0x7ff980083100 2021-03-09T10:24:40.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.233+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003ac40 con 0x7ff980083100 2021-03-09T10:24:40.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.233+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5882 cs=0 l=1 rev1=1 rx=0x7ff98008a290 tx=0x7ff98c00c770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:40.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.233+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c086390 con 0x7ff980083100 2021-03-09T10:24:40.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.237+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c099d50 con 0x7ff980083100 2021-03-09T10:24:40.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.237+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980083100 2021-03-09T10:24:40.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.237+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:40.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.241+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:40.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:40.241+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:45.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:45.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:45.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5882 cs=0 l=1 rev1=1 rx=0x7ff98008a290 tx=0x7ff98c00c770).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:45.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5882 cs=0 l=1 rev1=1 rx=0x7ff98008a290 tx=0x7ff98c00c770).stop 2021-03-09T10:24:45.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:45.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=CLOSED pgs=5882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:45.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:45.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:45.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:45.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053600 con 0x7ff99c0adc20 2021-03-09T10:24:45.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011120 con 0x7ff99c0613a0 2021-03-09T10:24:45.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001bb70 con 0x7ff980050140 2021-03-09T10:24:45.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:45.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:45.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:45.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:45.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.228+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:45.238 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:45.232+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19719 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:45.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.232+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:45.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.232+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:45.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.232+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:45.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.232+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:45.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.232+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a320 con 0x7ff980050140 2021-03-09T10:24:45.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.232+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003ac40 con 0x7ff980050140 2021-03-09T10:24:45.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.232+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5887 cs=0 l=1 rev1=1 rx=0x7ff98c017250 tx=0x7ff98c07a0e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:45.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.232+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a0130 con 0x7ff980050140 2021-03-09T10:24:45.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.232+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04bd70 con 0x7ff980050140 2021-03-09T10:24:45.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.236+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072970 con 0x7ff980050140 2021-03-09T10:24:45.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.236+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:45.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.240+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:45.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:45.240+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:50.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:50.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:50.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5887 cs=0 l=1 rev1=1 rx=0x7ff98c017250 tx=0x7ff98c07a0e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:50.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5887 cs=0 l=1 rev1=1 rx=0x7ff98c017250 tx=0x7ff98c07a0e0).stop 2021-03-09T10:24:50.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:50.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=CLOSED pgs=5887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:50.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:50.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:50.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:50.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:50.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053600 con 0x7ff99c0adc20 2021-03-09T10:24:50.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:50.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a320 con 0x7ff99c0613a0 2021-03-09T10:24:50.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045990 con 0x7ff980083100 2021-03-09T10:24:50.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:50.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:50.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:50.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:50.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:50.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:50.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:50.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045b00 con 0x7ff980083100 2021-03-09T10:24:50.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.232+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035f00 con 0x7ff980083100 2021-03-09T10:24:50.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.236+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5897 cs=0 l=1 rev1=1 rx=0x7ff994020730 tx=0x7ff994006bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:50.243 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:50.236+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19758 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:50.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.236+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095120 con 0x7ff980083100 2021-03-09T10:24:50.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.236+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095350 con 0x7ff980083100 2021-03-09T10:24:50.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.240+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:24:50.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.240+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:50.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.244+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:50.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:50.244+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:24:55.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:55.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:55.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5897 cs=0 l=1 rev1=1 rx=0x7ff994020730 tx=0x7ff994006bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:55.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5897 cs=0 l=1 rev1=1 rx=0x7ff994020730 tx=0x7ff994006bf0).stop 2021-03-09T10:24:55.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:55.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=CLOSED pgs=5897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:55.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:55.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:55.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:55.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a320 con 0x7ff99c0adc20 2021-03-09T10:24:55.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045940 con 0x7ff99c0613a0 2021-03-09T10:24:55.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003b0a0 con 0x7ff980050140 2021-03-09T10:24:55.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:24:55.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:55.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:55.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:24:55.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.236+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:55.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.240+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:55.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.240+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:55.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.240+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:55.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.240+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:55.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.240+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980038730 con 0x7ff980050140 2021-03-09T10:24:55.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.240+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045b00 con 0x7ff980050140 2021-03-09T10:24:55.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.240+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5909 cs=0 l=1 rev1=1 rx=0x7ff98002d9e0 tx=0x7ff994095120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:55.248 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:24:55.240+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19793 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:24:55.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.244+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff980050140 2021-03-09T10:24:55.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.244+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980050140 2021-03-09T10:24:55.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.248+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980050140 2021-03-09T10:24:55.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.248+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:55.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.248+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:55.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:24:55.248+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:00.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:00.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:00.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5909 cs=0 l=1 rev1=1 rx=0x7ff98002d9e0 tx=0x7ff994095120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:00.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5909 cs=0 l=1 rev1=1 rx=0x7ff98002d9e0 tx=0x7ff994095120).stop 2021-03-09T10:25:00.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:00.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=CLOSED pgs=5909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:00.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:00.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:00.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:00.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004cf30 con 0x7ff99c0adc20 2021-03-09T10:25:00.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d070 con 0x7ff99c0613a0 2021-03-09T10:25:00.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d1b0 con 0x7ff980083100 2021-03-09T10:25:00.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:00.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:00.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:00.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:00.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.240+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:00.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.244+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:00.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.244+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:00.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.244+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:00.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.244+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:00.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.244+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980010a90 con 0x7ff980083100 2021-03-09T10:25:00.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.244+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980038730 con 0x7ff980083100 2021-03-09T10:25:00.251 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:00.244+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19832 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:00.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.244+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5919 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff9940aa210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:00.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.244+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff980083100 2021-03-09T10:25:00.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.248+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994012df0 con 0x7ff980083100 2021-03-09T10:25:00.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.252+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff980083100 2021-03-09T10:25:00.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.252+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:00.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.252+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:00.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:00.252+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:01.796 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:25:01.799 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:25:05.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:05.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:05.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5919 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff9940aa210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:05.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5919 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff9940aa210).stop 2021-03-09T10:25:05.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:05.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=CLOSED pgs=5919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:05.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:05.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:05.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:05.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094b60 con 0x7ff99c0adc20 2021-03-09T10:25:05.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a320 con 0x7ff99c0613a0 2021-03-09T10:25:05.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004cf10 con 0x7ff980050140 2021-03-09T10:25:05.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:05.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:05.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:05.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:05.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.244+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:05.253 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:05.248+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19867 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:05.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.248+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:05.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.248+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:05.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.248+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:05.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.248+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:05.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.248+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004d080 con 0x7ff980050140 2021-03-09T10:25:05.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.248+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980010a90 con 0x7ff980050140 2021-03-09T10:25:05.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.252+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5934 cs=0 l=1 rev1=1 rx=0x7ff994078100 tx=0x7ff994009e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:05.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.252+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940835b0 con 0x7ff980050140 2021-03-09T10:25:05.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.252+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940835b0 con 0x7ff980050140 2021-03-09T10:25:05.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.256+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff980050140 2021-03-09T10:25:05.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.256+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:05.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.256+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:05.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:05.256+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:10.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:10.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:10.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5934 cs=0 l=1 rev1=1 rx=0x7ff994078100 tx=0x7ff994009e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:10.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5934 cs=0 l=1 rev1=1 rx=0x7ff994078100 tx=0x7ff994009e80).stop 2021-03-09T10:25:10.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:10.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=CLOSED pgs=5934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:10.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:10.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:10.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:10.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800098f0 con 0x7ff99c0adc20 2021-03-09T10:25:10.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009a30 con 0x7ff99c0613a0 2021-03-09T10:25:10.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980045940 con 0x7ff980083100 2021-03-09T10:25:10.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:10.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:10.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:10.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:10.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.248+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:10.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.252+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:10.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.252+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:10.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.252+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:10.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.252+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:10.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.252+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980045a80 con 0x7ff980083100 2021-03-09T10:25:10.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.252+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004d080 con 0x7ff980083100 2021-03-09T10:25:10.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.252+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5941 cs=0 l=1 rev1=1 rx=0x7ff9940aa3d0 tx=0x7ff9940293e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:10.259 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:10.252+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19903 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:10.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.252+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940411e0 con 0x7ff980083100 2021-03-09T10:25:10.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.252+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940411e0 con 0x7ff980083100 2021-03-09T10:25:10.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.256+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:25:10.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.256+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:10.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.260+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:10.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:10.260+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:15.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:15.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:15.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5941 cs=0 l=1 rev1=1 rx=0x7ff9940aa3d0 tx=0x7ff9940293e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:15.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5941 cs=0 l=1 rev1=1 rx=0x7ff9940aa3d0 tx=0x7ff9940293e0).stop 2021-03-09T10:25:15.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:15.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=CLOSED pgs=5941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:15.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:15.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:15.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:15.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093f40 con 0x7ff99c0adc20 2021-03-09T10:25:15.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014ac0 con 0x7ff99c0613a0 2021-03-09T10:25:15.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800337b0 con 0x7ff980050140 2021-03-09T10:25:15.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:15.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:15.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:15.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:15.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.252+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:15.262 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:15.256+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19942 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:15.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.256+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:15.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.256+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:15.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.256+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:15.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.256+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:15.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.256+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a320 con 0x7ff980050140 2021-03-09T10:25:15.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.256+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003ac40 con 0x7ff980050140 2021-03-09T10:25:15.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.260+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5955 cs=0 l=1 rev1=1 rx=0x7ff98c022f20 tx=0x7ff98c017120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:15.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c02cee0 con 0x7ff980050140 2021-03-09T10:25:15.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02cee0 con 0x7ff980050140 2021-03-09T10:25:15.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c09f170 con 0x7ff980050140 2021-03-09T10:25:15.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.264+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:15.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.264+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:15.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:15.264+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:20.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:20.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:20.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5955 cs=0 l=1 rev1=1 rx=0x7ff98c022f20 tx=0x7ff98c017120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:20.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5955 cs=0 l=1 rev1=1 rx=0x7ff98c022f20 tx=0x7ff98c017120).stop 2021-03-09T10:25:20.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:20.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=CLOSED pgs=5955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:20.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:20.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:20.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:20.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:20.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014aa0 con 0x7ff99c0adc20 2021-03-09T10:25:20.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:20.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093ef0 con 0x7ff99c0613a0 2021-03-09T10:25:20.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a320 con 0x7ff980083100 2021-03-09T10:25:20.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:20.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:20.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:20.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:20.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:20.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:20.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:20.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800098f0 con 0x7ff980083100 2021-03-09T10:25:20.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.256+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980045a80 con 0x7ff980083100 2021-03-09T10:25:20.368 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:20.260+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19977 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:20.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.260+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5963 cs=0 l=1 rev1=1 rx=0x7ff9940780d0 tx=0x7ff994098f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:20.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994084180 con 0x7ff980083100 2021-03-09T10:25:20.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994060770 con 0x7ff980083100 2021-03-09T10:25:20.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:25:20.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.268+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:20.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.268+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:20.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:20.268+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:25.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:25.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:25.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5963 cs=0 l=1 rev1=1 rx=0x7ff9940780d0 tx=0x7ff994098f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:25.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 secure :-1 s=READY pgs=5963 cs=0 l=1 rev1=1 rx=0x7ff9940780d0 tx=0x7ff994098f00).stop 2021-03-09T10:25:25.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980053110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:25.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980053110 unknown :-1 s=CLOSED pgs=5963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:25.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:25.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:25.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:25.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a320 con 0x7ff99c0adc20 2021-03-09T10:25:25.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005a40 con 0x7ff99c0613a0 2021-03-09T10:25:25.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800551d0 con 0x7ff980050140 2021-03-09T10:25:25.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:25.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:25.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:25.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:25.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.260+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:25.270 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:25.264+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20016 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:25.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:25.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:25.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:25.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:25.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088c80 con 0x7ff980050140 2021-03-09T10:25:25.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003ac40 con 0x7ff980050140 2021-03-09T10:25:25.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5977 cs=0 l=1 rev1=1 rx=0x7ff98c0798c0 tx=0x7ff98c003f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:25.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980050140 2021-03-09T10:25:25.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980050140 2021-03-09T10:25:25.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980050140 2021-03-09T10:25:25.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.272+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:25.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.272+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:25.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:25.272+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:30.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:30.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:30.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5977 cs=0 l=1 rev1=1 rx=0x7ff98c0798c0 tx=0x7ff98c003f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:30.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5977 cs=0 l=1 rev1=1 rx=0x7ff98c0798c0 tx=0x7ff98c003f60).stop 2021-03-09T10:25:30.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:30.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=CLOSED pgs=5977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:30.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:30.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:30.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:30.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016ef0 con 0x7ff99c0adc20 2021-03-09T10:25:30.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055180 con 0x7ff99c0613a0 2021-03-09T10:25:30.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800552c0 con 0x7ff980083100 2021-03-09T10:25:30.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:30.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:30.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:30.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:30.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:30.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:30.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:30.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:30.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:30.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980053110 con 0x7ff980083100 2021-03-09T10:25:30.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.264+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800098f0 con 0x7ff980083100 2021-03-09T10:25:30.276 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:30.268+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20051 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:30.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.268+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005a40 secure :-1 s=READY pgs=5983 cs=0 l=1 rev1=1 rx=0x7ff9940b2050 tx=0x7ff99406df10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:30.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409bbd0 con 0x7ff980083100 2021-03-09T10:25:30.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b9c0 con 0x7ff980083100 2021-03-09T10:25:30.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:25:30.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.272+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:30.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.272+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:30.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:30.272+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:31.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:25:31.844 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:25:35.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:25:35.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:35.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005a40 secure :-1 s=READY pgs=5983 cs=0 l=1 rev1=1 rx=0x7ff9940b2050 tx=0x7ff99406df10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:35.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005a40 secure :-1 s=READY pgs=5983 cs=0 l=1 rev1=1 rx=0x7ff9940b2050 tx=0x7ff99406df10).stop 2021-03-09T10:25:35.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:35.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005a40 unknown :-1 s=CLOSED pgs=5983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:35.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:35.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:35.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:35.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a320 con 0x7ff99c0adc20 2021-03-09T10:25:35.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009990 con 0x7ff99c0613a0 2021-03-09T10:25:35.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009ad0 con 0x7ff980050140 2021-03-09T10:25:35.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:35.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:35.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:35.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:35.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.268+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:35.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.272+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:35.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.272+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:35.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.272+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:35.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.272+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:35.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.272+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054fe0 con 0x7ff980050140 2021-03-09T10:25:35.277 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:35.272+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20090 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:35.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.272+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980053110 con 0x7ff980050140 2021-03-09T10:25:35.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.272+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5998 cs=0 l=1 rev1=1 rx=0x7ff994078100 tx=0x7ff99409bb20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:35.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405f1c0 con 0x7ff980050140 2021-03-09T10:25:35.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405f530 con 0x7ff980050140 2021-03-09T10:25:35.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980050140 2021-03-09T10:25:35.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.280+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:35.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.280+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:35.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:35.280+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:40.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:40.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:40.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5998 cs=0 l=1 rev1=1 rx=0x7ff994078100 tx=0x7ff99409bb20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:40.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 secure :-1 s=READY pgs=5998 cs=0 l=1 rev1=1 rx=0x7ff994078100 tx=0x7ff99409bb20).stop 2021-03-09T10:25:40.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:40.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980039e30 unknown :-1 s=CLOSED pgs=5998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:40.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:40.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:40.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:40.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055170 con 0x7ff99c0adc20 2021-03-09T10:25:40.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800552b0 con 0x7ff99c0613a0 2021-03-09T10:25:40.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053110 con 0x7ff980083100 2021-03-09T10:25:40.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:40.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:40.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:40.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:40.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.272+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:40.281 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:40.276+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20125 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:40.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T10:25:40.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:40.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980029210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:40.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980029210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:40.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980021c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:40.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980021c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:40.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff9780053d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:40.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff978007970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:40.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff978009ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:40.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055170 con 0x7ff99c0adc20 2021-03-09T10:25:40.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800552b0 con 0x7ff99c0613a0 2021-03-09T10:25:40.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053110 con 0x7ff980083100 2021-03-09T10:25:40.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff978007970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:40.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff9780053d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:40.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff978007970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:40.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.276+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff9780053d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:40.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.280+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff978009ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:40.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff978007970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:40.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff978007970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:40.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff9780053d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:40.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff9780053d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:40.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980053250 con 0x7ff980083100 2021-03-09T10:25:40.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054fe0 con 0x7ff980083100 2021-03-09T10:25:40.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff978009ef0 secure :-1 s=READY pgs=6013 cs=0 l=1 rev1=1 rx=0x7ff994081840 tx=0x7ff994060d70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:40.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980083100 2021-03-09T10:25:40.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994036a50 con 0x7ff980083100 2021-03-09T10:25:40.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:25:40.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.288+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:40.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.288+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:40.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:40.288+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:45.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff978009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:45.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff978009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:45.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff978009ef0 secure :-1 s=READY pgs=6013 cs=0 l=1 rev1=1 rx=0x7ff994081840 tx=0x7ff994060d70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:45.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff978009ef0 secure :-1 s=READY pgs=6013 cs=0 l=1 rev1=1 rx=0x7ff994081840 tx=0x7ff994060d70).stop 2021-03-09T10:25:45.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff978009ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:45.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff978009ef0 unknown :-1 s=CLOSED pgs=6013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:45.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:45.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:45.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:45.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003ae30 con 0x7ff99c0adc20 2021-03-09T10:25:45.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a320 con 0x7ff99c0613a0 2021-03-09T10:25:45.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff980050140 2021-03-09T10:25:45.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:45.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:45.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:45.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:45.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.276+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:45.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:45.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:45.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:45.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:45.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020ae0 con 0x7ff980050140 2021-03-09T10:25:45.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088c80 con 0x7ff980050140 2021-03-09T10:25:45.288 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:45.280+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20164 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:45.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 secure :-1 s=READY pgs=6016 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c00bb80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:45.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a9120 con 0x7ff980050140 2021-03-09T10:25:45.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980050140 2021-03-09T10:25:45.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a9120 con 0x7ff980050140 2021-03-09T10:25:45.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.284+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:45.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.288+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:45.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:45.288+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:50.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:25:50.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:50.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 secure :-1 s=READY pgs=6016 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c00bb80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:50.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 secure :-1 s=READY pgs=6016 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c00bb80).stop 2021-03-09T10:25:50.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980005a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:50.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 unknown :-1 s=CLOSED pgs=6016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:50.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:50.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:50.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:50.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c90 con 0x7ff99c0adc20 2021-03-09T10:25:50.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036230 con 0x7ff99c0613a0 2021-03-09T10:25:50.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003ae30 con 0x7ff980083100 2021-03-09T10:25:50.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:50.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:50.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:50.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:50.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.280+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:50.290 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:50.284+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20199 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:50.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:50.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:50.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:50.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:50.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088c80 con 0x7ff980083100 2021-03-09T10:25:50.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980053250 con 0x7ff980083100 2021-03-09T10:25:50.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 secure :-1 s=READY pgs=6025 cs=0 l=1 rev1=1 rx=0x7ff994069d70 tx=0x7ff9940011f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:50.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980083100 2021-03-09T10:25:50.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:25:50.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:25:50.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.292+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:50.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.292+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:50.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:50.292+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:25:55.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980029170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:55.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980029170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:55.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 secure :-1 s=READY pgs=6025 cs=0 l=1 rev1=1 rx=0x7ff994069d70 tx=0x7ff9940011f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:55.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 secure :-1 s=READY pgs=6025 cs=0 l=1 rev1=1 rx=0x7ff994069d70 tx=0x7ff9940011f0).stop 2021-03-09T10:25:55.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980029170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:55.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 unknown :-1 s=CLOSED pgs=6025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:55.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:55.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:55.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:55.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:25:55.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:55.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c90 con 0x7ff99c0adc20 2021-03-09T10:25:55.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092010 con 0x7ff99c0613a0 2021-03-09T10:25:55.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036230 con 0x7ff980050140 2021-03-09T10:25:55.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:25:55.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:55.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:55.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:55.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:55.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:55.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:55.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff980050140 2021-03-09T10:25:55.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.284+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020ae0 con 0x7ff980050140 2021-03-09T10:25:55.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.288+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 secure :-1 s=READY pgs=6037 cs=0 l=1 rev1=1 rx=0x7ff980011300 tx=0x7ff98c006780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:55.295 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:25:55.288+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20238 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:25:55.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.288+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a9330 con 0x7ff980050140 2021-03-09T10:25:55.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980050140 2021-03-09T10:25:55.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.292+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980050140 2021-03-09T10:25:55.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.292+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:55.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.296+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:55.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:25:55.296+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:00.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:00.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:00.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 secure :-1 s=READY pgs=6037 cs=0 l=1 rev1=1 rx=0x7ff980011300 tx=0x7ff98c006780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:00.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 secure :-1 s=READY pgs=6037 cs=0 l=1 rev1=1 rx=0x7ff980011300 tx=0x7ff98c006780).stop 2021-03-09T10:26:00.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980005a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:00.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 unknown :-1 s=CLOSED pgs=6037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:00.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:00.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:00.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:00.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030340 con 0x7ff99c0adc20 2021-03-09T10:26:00.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036230 con 0x7ff99c0613a0 2021-03-09T10:26:00.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092160 con 0x7ff980083100 2021-03-09T10:26:00.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:00.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:00.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:00.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:00.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.287+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:00.297 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:00.291+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20273 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:00.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:00.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:00.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:00.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:00.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:26:00.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088c80 con 0x7ff980083100 2021-03-09T10:26:00.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 secure :-1 s=READY pgs=6049 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940aa080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:00.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994041580 con 0x7ff980083100 2021-03-09T10:26:00.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940835b0 con 0x7ff980083100 2021-03-09T10:26:00.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:26:00.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.299+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:00.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.299+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:00.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:00.299+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:01.895 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:26:01.898 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:26:05.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980029170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:05.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980029170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:05.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 secure :-1 s=READY pgs=6049 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940aa080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:05.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 secure :-1 s=READY pgs=6049 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940aa080).stop 2021-03-09T10:26:05.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980029170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:05.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980029170 unknown :-1 s=CLOSED pgs=6049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:05.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:05.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:05.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:05.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800304e0 con 0x7ff99c0adc20 2021-03-09T10:26:05.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092160 con 0x7ff99c0613a0 2021-03-09T10:26:05.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff980050140 2021-03-09T10:26:05.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:05.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:05.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:05.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:05.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:05.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:05.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:05.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:05.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:05.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005f30 con 0x7ff980050140 2021-03-09T10:26:05.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.295+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088c80 con 0x7ff980050140 2021-03-09T10:26:05.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.295+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 secure :-1 s=READY pgs=6057 cs=0 l=1 rev1=1 rx=0x7ff99400bf80 tx=0x7ff99405dac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:05.304 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:05.295+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20312 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:05.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980050140 2021-03-09T10:26:05.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980050140 2021-03-09T10:26:05.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994069860 con 0x7ff980050140 2021-03-09T10:26:05.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.299+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:05.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.299+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:05.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:05.299+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:10.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:10.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:10.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 secure :-1 s=READY pgs=6057 cs=0 l=1 rev1=1 rx=0x7ff99400bf80 tx=0x7ff99405dac0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:10.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 secure :-1 s=READY pgs=6057 cs=0 l=1 rev1=1 rx=0x7ff99400bf80 tx=0x7ff99405dac0).stop 2021-03-09T10:26:10.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980005a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:10.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980005a40 unknown :-1 s=CLOSED pgs=6057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:10.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:10.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:10.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:10.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800304e0 con 0x7ff99c0adc20 2021-03-09T10:26:10.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092160 con 0x7ff99c0613a0 2021-03-09T10:26:10.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005f30 con 0x7ff980083100 2021-03-09T10:26:10.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:10.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:10.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:10.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:10.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.295+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:10.305 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:10.299+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20347 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:10.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:10.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:10.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:10.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:10.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980029640 con 0x7ff980083100 2021-03-09T10:26:10.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088c80 con 0x7ff980083100 2021-03-09T10:26:10.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6074 cs=0 l=1 rev1=1 rx=0x7ff994095630 tx=0x7ff99406a090).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:10.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994075110 con 0x7ff980083100 2021-03-09T10:26:10.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994075110 con 0x7ff980083100 2021-03-09T10:26:10.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700c0 con 0x7ff980083100 2021-03-09T10:26:10.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.307+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:10.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.307+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:10.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:10.307+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:15.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:15.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:15.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6074 cs=0 l=1 rev1=1 rx=0x7ff994095630 tx=0x7ff99406a090).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:15.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6074 cs=0 l=1 rev1=1 rx=0x7ff994095630 tx=0x7ff99406a090).stop 2021-03-09T10:26:15.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:15.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=CLOSED pgs=6074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:15.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:15.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:15.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:15.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ed80 con 0x7ff99c0adc20 2021-03-09T10:26:15.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:15.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f410 con 0x7ff99c0613a0 2021-03-09T10:26:15.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095250 con 0x7ff980050140 2021-03-09T10:26:15.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:15.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:15.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:15.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:15.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:15.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:15.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:15.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:15.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980088c80 con 0x7ff980050140 2021-03-09T10:26:15.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980029640 con 0x7ff980050140 2021-03-09T10:26:15.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092d50 secure :-1 s=READY pgs=6079 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99407a200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:15.313 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:15.303+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20386 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:15.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408e890 con 0x7ff980050140 2021-03-09T10:26:15.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408e6b0 con 0x7ff980050140 2021-03-09T10:26:15.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980050140 2021-03-09T10:26:15.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.311+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:15.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.311+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:15.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:15.311+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:20.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:26:20.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:20.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092d50 secure :-1 s=READY pgs=6079 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99407a200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:20.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092d50 secure :-1 s=READY pgs=6079 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99407a200).stop 2021-03-09T10:26:20.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 msgr2=0x7ff980092d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:20.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980050140 0x7ff980092d50 unknown :-1 s=CLOSED pgs=6079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:20.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:20.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:20.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:20.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095250 con 0x7ff99c0adc20 2021-03-09T10:26:20.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ed80 con 0x7ff99c0613a0 2021-03-09T10:26:20.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f560 con 0x7ff980083100 2021-03-09T10:26:20.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:20.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:20.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:20.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:20.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:20.314 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:20.307+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20421 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:20.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:20.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:20.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:20.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:20.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800060e0 con 0x7ff980083100 2021-03-09T10:26:20.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980088c80 con 0x7ff980083100 2021-03-09T10:26:20.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.311+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6093 cs=0 l=1 rev1=1 rx=0x7ff9940aa130 tx=0x7ff994083620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:20.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994034470 con 0x7ff980083100 2021-03-09T10:26:20.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:26:20.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:26:20.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.319+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:20.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.319+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:20.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:20.319+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:25.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:25.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:25.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6093 cs=0 l=1 rev1=1 rx=0x7ff9940aa130 tx=0x7ff994083620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:25.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6093 cs=0 l=1 rev1=1 rx=0x7ff9940aa130 tx=0x7ff994083620).stop 2021-03-09T10:26:25.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:25.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=CLOSED pgs=6093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:25.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:25.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:25.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:25.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095250 con 0x7ff99c0adc20 2021-03-09T10:26:25.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a810 con 0x7ff99c0613a0 2021-03-09T10:26:25.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088c80 con 0x7ff9800301a0 2021-03-09T10:26:25.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:25.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:25.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:25.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:25.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.307+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:25.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:25.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:25.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:25.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:25.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800399c0 con 0x7ff9800301a0 2021-03-09T10:26:25.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020ae0 con 0x7ff9800301a0 2021-03-09T10:26:25.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6101 cs=0 l=1 rev1=1 rx=0x7ff9800216b0 tx=0x7ff98c079700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:25.321 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:25.311+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20459 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:25.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff9800301a0 2021-03-09T10:26:25.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079fc0 con 0x7ff9800301a0 2021-03-09T10:26:25.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff9800301a0 2021-03-09T10:26:25.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.315+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:25.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.319+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:25.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:25.319+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:30.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:30.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:30.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6101 cs=0 l=1 rev1=1 rx=0x7ff9800216b0 tx=0x7ff98c079700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:30.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6101 cs=0 l=1 rev1=1 rx=0x7ff9800216b0 tx=0x7ff98c079700).stop 2021-03-09T10:26:30.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:30.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=CLOSED pgs=6101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:30.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:30.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:30.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:30.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095250 con 0x7ff99c0adc20 2021-03-09T10:26:30.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088c80 con 0x7ff99c0613a0 2021-03-09T10:26:30.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a7d0 con 0x7ff980083100 2021-03-09T10:26:30.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:30.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:30.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:30.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:30.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.311+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:30.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:30.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:30.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:30.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:30.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092160 con 0x7ff980083100 2021-03-09T10:26:30.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800060e0 con 0x7ff980083100 2021-03-09T10:26:30.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6111 cs=0 l=1 rev1=1 rx=0x7ff9940aa240 tx=0x7ff994029120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:30.325 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:30.315+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20495 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:30.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994098120 con 0x7ff980083100 2021-03-09T10:26:30.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:26:30.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:26:30.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.319+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:30.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.323+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:30.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:30.323+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:31.976 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:26:31.992 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:26:35.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:35.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:35.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6111 cs=0 l=1 rev1=1 rx=0x7ff9940aa240 tx=0x7ff994029120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:35.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6111 cs=0 l=1 rev1=1 rx=0x7ff9940aa240 tx=0x7ff994029120).stop 2021-03-09T10:26:35.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:35.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=CLOSED pgs=6111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:35.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:35.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:35.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:35.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095250 con 0x7ff99c0adc20 2021-03-09T10:26:35.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046900 con 0x7ff99c0613a0 2021-03-09T10:26:35.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035a90 con 0x7ff9800301a0 2021-03-09T10:26:35.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:35.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:35.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:35.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:35.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:35.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:35.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:35.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:35.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:35.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff9800301a0 2021-03-09T10:26:35.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092160 con 0x7ff9800301a0 2021-03-09T10:26:35.329 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:35.319+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20533 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:35.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.319+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6121 cs=0 l=1 rev1=1 rx=0x7ff980085a60 tx=0x7ff994020d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:35.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940413c0 con 0x7ff9800301a0 2021-03-09T10:26:35.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940411e0 con 0x7ff9800301a0 2021-03-09T10:26:35.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060db0 con 0x7ff9800301a0 2021-03-09T10:26:35.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.327+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:35.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.331+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:35.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:35.331+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:40.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:40.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:40.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6121 cs=0 l=1 rev1=1 rx=0x7ff980085a60 tx=0x7ff994020d50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:40.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6121 cs=0 l=1 rev1=1 rx=0x7ff980085a60 tx=0x7ff994020d50).stop 2021-03-09T10:26:40.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:40.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=CLOSED pgs=6121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:40.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:40.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:40.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:40.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980095230 con 0x7ff99c0adc20 2021-03-09T10:26:40.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035a90 con 0x7ff99c0613a0 2021-03-09T10:26:40.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:40.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:40.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:26:40.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:40.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:40.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:40.332 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:40.323+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20569 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:40.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:40.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:40.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:40.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:40.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800888f0 con 0x7ff980083100 2021-03-09T10:26:40.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092160 con 0x7ff980083100 2021-03-09T10:26:40.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6134 cs=0 l=1 rev1=1 rx=0x7ff994009bb0 tx=0x7ff9940843d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:40.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940411e0 con 0x7ff980083100 2021-03-09T10:26:40.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940413c0 con 0x7ff980083100 2021-03-09T10:26:40.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994034a10 con 0x7ff980083100 2021-03-09T10:26:40.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.331+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:40.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.335+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:40.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:40.335+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:45.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:26:45.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:45.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6134 cs=0 l=1 rev1=1 rx=0x7ff994009bb0 tx=0x7ff9940843d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:45.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6134 cs=0 l=1 rev1=1 rx=0x7ff994009bb0 tx=0x7ff9940843d0).stop 2021-03-09T10:26:45.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:45.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=CLOSED pgs=6134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:45.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:45.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:45.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:45.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003f540 con 0x7ff99c0adc20 2021-03-09T10:26:45.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f200 con 0x7ff99c0613a0 2021-03-09T10:26:45.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff9800301a0 2021-03-09T10:26:45.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:45.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:45.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:45.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:45.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:45.335 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:45.327+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20607 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:45.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:45.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:45.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:45.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:45.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035a90 con 0x7ff9800301a0 2021-03-09T10:26:45.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.327+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800888f0 con 0x7ff9800301a0 2021-03-09T10:26:45.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6147 cs=0 l=1 rev1=1 rx=0x7ff994034e50 tx=0x7ff99409df80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:45.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081690 con 0x7ff9800301a0 2021-03-09T10:26:45.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081690 con 0x7ff9800301a0 2021-03-09T10:26:45.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff9800301a0 2021-03-09T10:26:45.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.339+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:45.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.339+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:45.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:45.339+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:50.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:50.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:50.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6147 cs=0 l=1 rev1=1 rx=0x7ff994034e50 tx=0x7ff99409df80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:50.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6147 cs=0 l=1 rev1=1 rx=0x7ff994034e50 tx=0x7ff99409df80).stop 2021-03-09T10:26:50.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:50.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=CLOSED pgs=6147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:50.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:50.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:50.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:50.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e550 con 0x7ff99c0adc20 2021-03-09T10:26:50.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091e20 con 0x7ff99c0613a0 2021-03-09T10:26:50.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980091f60 con 0x7ff980083100 2021-03-09T10:26:50.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:50.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:50.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:50.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:50.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:50.338 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:50.331+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20643 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:50.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:50.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:50.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:50.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:50.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800920a0 con 0x7ff980083100 2021-03-09T10:26:50.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035a90 con 0x7ff980083100 2021-03-09T10:26:50.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6155 cs=0 l=1 rev1=1 rx=0x7ff99406d590 tx=0x7ff9940415b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:50.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081690 con 0x7ff980083100 2021-03-09T10:26:50.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081690 con 0x7ff980083100 2021-03-09T10:26:50.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:26:50.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.339+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:50.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.343+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:50.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:50.343+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:26:55.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:55.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:55.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6155 cs=0 l=1 rev1=1 rx=0x7ff99406d590 tx=0x7ff9940415b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:55.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6155 cs=0 l=1 rev1=1 rx=0x7ff99406d590 tx=0x7ff9940415b0).stop 2021-03-09T10:26:55.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:55.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=CLOSED pgs=6155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:55.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:55.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:55.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:55.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800574b0 con 0x7ff99c0adc20 2021-03-09T10:26:55.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e550 con 0x7ff99c0613a0 2021-03-09T10:26:55.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980035a90 con 0x7ff9800301a0 2021-03-09T10:26:55.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:26:55.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:55.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:26:55.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:55.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:55.342 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:26:55.335+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20681 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:26:55.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:55.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:55.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:55.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:55.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff9800301a0 2021-03-09T10:26:55.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800399c0 con 0x7ff9800301a0 2021-03-09T10:26:55.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.335+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6166 cs=0 l=1 rev1=1 rx=0x7ff98003e1b0 tx=0x7ff98c079ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:55.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04bec0 con 0x7ff9800301a0 2021-03-09T10:26:55.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.343+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04bec0 con 0x7ff9800301a0 2021-03-09T10:26:55.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.347+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072970 con 0x7ff9800301a0 2021-03-09T10:26:55.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.347+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:55.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.347+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:55.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:26:55.347+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:00.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:00.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:00.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6166 cs=0 l=1 rev1=1 rx=0x7ff98003e1b0 tx=0x7ff98c079ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:00.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 secure :-1 s=READY pgs=6166 cs=0 l=1 rev1=1 rx=0x7ff98003e1b0 tx=0x7ff98c079ed0).stop 2021-03-09T10:27:00.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:00.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d50 unknown :-1 s=CLOSED pgs=6166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:00.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:00.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:00.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:00.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800574b0 con 0x7ff99c0adc20 2021-03-09T10:27:00.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c730 con 0x7ff99c0613a0 2021-03-09T10:27:00.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094ba0 con 0x7ff980083100 2021-03-09T10:27:00.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:00.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:00.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:00.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:00.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:00.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:00.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:00.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:00.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:00.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008e550 con 0x7ff980083100 2021-03-09T10:27:00.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800920a0 con 0x7ff980083100 2021-03-09T10:27:00.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6175 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99405ffd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:00.348 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:00.339+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20717 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:00.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940130c0 con 0x7ff980083100 2021-03-09T10:27:00.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940349d0 con 0x7ff980083100 2021-03-09T10:27:00.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.347+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994020dc0 con 0x7ff980083100 2021-03-09T10:27:00.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.347+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:00.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.347+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:00.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:00.347+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:02.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:27:02.060 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:27:05.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:05.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:05.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6175 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99405ffd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:05.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 secure :-1 s=READY pgs=6175 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99405ffd0).stop 2021-03-09T10:27:05.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800394d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:05.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800394d0 unknown :-1 s=CLOSED pgs=6175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:05.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:05.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:05.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:05.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a7a0 con 0x7ff99c0adc20 2021-03-09T10:27:05.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094d00 con 0x7ff99c0613a0 2021-03-09T10:27:05.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c730 con 0x7ff9800301a0 2021-03-09T10:27:05.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:05.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:05.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:05.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:05.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:05.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:05.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:05.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:05.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:05.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980035a90 con 0x7ff9800301a0 2021-03-09T10:27:05.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008e550 con 0x7ff9800301a0 2021-03-09T10:27:05.352 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:05.343+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20755 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:05.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 secure :-1 s=READY pgs=6185 cs=0 l=1 rev1=1 rx=0x7ff994013960 tx=0x7ff99400c280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:05.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.347+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994034c00 con 0x7ff9800301a0 2021-03-09T10:27:05.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.347+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff9800301a0 2021-03-09T10:27:05.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.351+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff9800301a0 2021-03-09T10:27:05.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.351+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:05.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.351+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:05.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:05.351+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:10.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:10.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:10.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 secure :-1 s=READY pgs=6185 cs=0 l=1 rev1=1 rx=0x7ff994013960 tx=0x7ff99400c280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:10.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 secure :-1 s=READY pgs=6185 cs=0 l=1 rev1=1 rx=0x7ff994013960 tx=0x7ff99400c280).stop 2021-03-09T10:27:10.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:10.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 unknown :-1 s=CLOSED pgs=6185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:10.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:10.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:10.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:10.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094d00 con 0x7ff99c0adc20 2021-03-09T10:27:10.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e550 con 0x7ff99c0613a0 2021-03-09T10:27:10.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a600 con 0x7ff980083100 2021-03-09T10:27:10.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:10.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:10.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:10.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:10.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.343+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:10.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.347+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:10.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.347+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:10.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.347+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:10.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.347+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:10.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.347+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a770 con 0x7ff980083100 2021-03-09T10:27:10.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.347+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980035a90 con 0x7ff980083100 2021-03-09T10:27:10.356 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:10.347+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20791 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:10.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.347+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6200 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99406a040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:10.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.351+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994013990 con 0x7ff980083100 2021-03-09T10:27:10.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.351+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994034b90 con 0x7ff980083100 2021-03-09T10:27:10.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.355+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994034b90 con 0x7ff980083100 2021-03-09T10:27:10.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.355+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:10.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.355+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:10.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:10.355+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:15.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:15.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:15.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6200 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99406a040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:15.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6200 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99406a040).stop 2021-03-09T10:27:15.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:15.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=CLOSED pgs=6200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:15.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:15.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:15.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:15.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c110 con 0x7ff99c0adc20 2021-03-09T10:27:15.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094d00 con 0x7ff99c0613a0 2021-03-09T10:27:15.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e550 con 0x7ff9800301a0 2021-03-09T10:27:15.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:15.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:15.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:15.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.346+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:15.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:15.359 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:15.350+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20829 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:15.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:15.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:15.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:15.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:15.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff9800301a0 2021-03-09T10:27:15.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003a770 con 0x7ff9800301a0 2021-03-09T10:27:15.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 secure :-1 s=READY pgs=6212 cs=0 l=1 rev1=1 rx=0x7ff99402a320 tx=0x7ff99400c190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:15.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020630 con 0x7ff9800301a0 2021-03-09T10:27:15.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994020630 con 0x7ff9800301a0 2021-03-09T10:27:15.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940411e0 con 0x7ff9800301a0 2021-03-09T10:27:15.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.362+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:15.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.366+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:15.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:15.366+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:20.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:20.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:20.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 secure :-1 s=READY pgs=6212 cs=0 l=1 rev1=1 rx=0x7ff99402a320 tx=0x7ff99400c190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:20.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 secure :-1 s=READY pgs=6212 cs=0 l=1 rev1=1 rx=0x7ff99402a320 tx=0x7ff99400c190).stop 2021-03-09T10:27:20.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:20.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 unknown :-1 s=CLOSED pgs=6212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:20.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:20.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:20.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:20.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:20.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:20.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094d00 con 0x7ff99c0adc20 2021-03-09T10:27:20.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e550 con 0x7ff99c0613a0 2021-03-09T10:27:20.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:27:20.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:20.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:20.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:20.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:20.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:20.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:20.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:20.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093900 con 0x7ff980083100 2021-03-09T10:27:20.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800399c0 con 0x7ff980083100 2021-03-09T10:27:20.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6217 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c04bbd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:20.365 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:20.354+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20865 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:20.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08e080 con 0x7ff980083100 2021-03-09T10:27:20.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00b120 con 0x7ff980083100 2021-03-09T10:27:20.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00b290 con 0x7ff980083100 2021-03-09T10:27:20.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.362+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:20.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.362+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:20.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:20.362+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:25.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:25.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:25.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6217 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c04bbd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:25.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6217 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c04bbd0).stop 2021-03-09T10:27:25.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:25.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=CLOSED pgs=6217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:25.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:25.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:25.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:25.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e5a0 con 0x7ff99c0adc20 2021-03-09T10:27:25.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0613a0 2021-03-09T10:27:25.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094b30 con 0x7ff9800301a0 2021-03-09T10:27:25.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:25.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:25.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.354+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:25.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:25.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.358+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:25.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.358+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:25.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.358+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:25.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.358+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:25.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.358+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:25.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.358+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094c70 con 0x7ff9800301a0 2021-03-09T10:27:25.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.358+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093900 con 0x7ff9800301a0 2021-03-09T10:27:25.369 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:25.358+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20903 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:25.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.358+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 secure :-1 s=READY pgs=6230 cs=0 l=1 rev1=1 rx=0x7ff98c07cdf0 tx=0x7ff98c017120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:25.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00b400 con 0x7ff9800301a0 2021-03-09T10:27:25.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00b400 con 0x7ff9800301a0 2021-03-09T10:27:25.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072970 con 0x7ff9800301a0 2021-03-09T10:27:25.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.366+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:25.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.370+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:25.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:25.370+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:30.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:30.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:30.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 secure :-1 s=READY pgs=6230 cs=0 l=1 rev1=1 rx=0x7ff98c07cdf0 tx=0x7ff98c017120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:30.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 secure :-1 s=READY pgs=6230 cs=0 l=1 rev1=1 rx=0x7ff98c07cdf0 tx=0x7ff98c017120).stop 2021-03-09T10:27:30.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:30.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175f0 unknown :-1 s=CLOSED pgs=6230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:30.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:30.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:30.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:30.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e5a0 con 0x7ff99c0adc20 2021-03-09T10:27:30.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009b80 con 0x7ff99c0613a0 2021-03-09T10:27:30.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094e00 con 0x7ff980083100 2021-03-09T10:27:30.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:30.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:30.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:30.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:30.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:30.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:30.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:30.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:30.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:30.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093900 con 0x7ff980083100 2021-03-09T10:27:30.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094c70 con 0x7ff980083100 2021-03-09T10:27:30.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.362+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6239 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0aa010).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:30.373 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:30.362+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20939 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:30.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a9910 con 0x7ff980083100 2021-03-09T10:27:30.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a9910 con 0x7ff980083100 2021-03-09T10:27:30.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bc40 con 0x7ff980083100 2021-03-09T10:27:30.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.370+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:30.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.370+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:30.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:30.370+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:32.124 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:27:32.127 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:27:35.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:35.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:35.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6239 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0aa010).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:35.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6239 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0aa010).stop 2021-03-09T10:27:35.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:35.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=CLOSED pgs=6239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:35.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:35.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:35.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:35.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0adc20 2021-03-09T10:27:35.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094c70 con 0x7ff99c0613a0 2021-03-09T10:27:35.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094db0 con 0x7ff9800301a0 2021-03-09T10:27:35.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:35.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:35.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:35.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:35.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:35.375 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:35.366+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20977 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:35.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:35.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:35.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:35.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:35.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980038c20 con 0x7ff9800301a0 2021-03-09T10:27:35.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093900 con 0x7ff9800301a0 2021-03-09T10:27:35.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.370+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175c0 secure :-1 s=READY pgs=6254 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c00bef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:35.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c03c0b0 con 0x7ff9800301a0 2021-03-09T10:27:35.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03c0b0 con 0x7ff9800301a0 2021-03-09T10:27:35.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff9800301a0 2021-03-09T10:27:35.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.378+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:35.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.378+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:35.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:35.378+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:40.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:40.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:40.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175c0 secure :-1 s=READY pgs=6254 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c00bef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:40.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175c0 secure :-1 s=READY pgs=6254 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c00bef0).stop 2021-03-09T10:27:40.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800175c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:40.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800175c0 unknown :-1 s=CLOSED pgs=6254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:40.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:40.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:40.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:40.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026bc0 con 0x7ff99c0adc20 2021-03-09T10:27:40.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002adb0 con 0x7ff99c0613a0 2021-03-09T10:27:40.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002aef0 con 0x7ff980083100 2021-03-09T10:27:40.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:40.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:40.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:40.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:40.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:40.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:40.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:40.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:40.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:40.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092b60 con 0x7ff980083100 2021-03-09T10:27:40.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.370+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980038c20 con 0x7ff980083100 2021-03-09T10:27:40.381 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:40.370+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21013 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:40.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6262 cs=0 l=1 rev1=1 rx=0x7ff98c07f040 tx=0x7ff98c0a9dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:40.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff980083100 2021-03-09T10:27:40.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff980083100 2021-03-09T10:27:40.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a31a0 con 0x7ff980083100 2021-03-09T10:27:40.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.378+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:40.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.382+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:40.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:40.382+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:45.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:45.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:45.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6262 cs=0 l=1 rev1=1 rx=0x7ff98c07f040 tx=0x7ff98c0a9dd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:45.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 secure :-1 s=READY pgs=6262 cs=0 l=1 rev1=1 rx=0x7ff98c07f040 tx=0x7ff98c0a9dd0).stop 2021-03-09T10:27:45.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980038730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:45.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980038730 unknown :-1 s=CLOSED pgs=6262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:45.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:45.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:45.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:45.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026b70 con 0x7ff99c0adc20 2021-03-09T10:27:45.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800399c0 con 0x7ff99c0613a0 2021-03-09T10:27:45.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff9800301a0 2021-03-09T10:27:45.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:45.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:45.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:45.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:45.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.374+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:45.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.378+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:45.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.378+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:45.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.378+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:45.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.378+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:45.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.378+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017b40 con 0x7ff9800301a0 2021-03-09T10:27:45.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.378+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003a770 con 0x7ff9800301a0 2021-03-09T10:27:45.386 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:45.378+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21051 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:45.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.378+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092cf0 secure :-1 s=READY pgs=6274 cs=0 l=1 rev1=1 rx=0x7ff99408e920 tx=0x7ff99407a1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:45.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940aa080 con 0x7ff9800301a0 2021-03-09T10:27:45.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408e950 con 0x7ff9800301a0 2021-03-09T10:27:45.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.382+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff9800301a0 2021-03-09T10:27:45.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.382+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:45.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.386+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:45.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:45.386+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:50.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:50.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:50.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092cf0 secure :-1 s=READY pgs=6274 cs=0 l=1 rev1=1 rx=0x7ff99408e920 tx=0x7ff99407a1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:50.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092cf0 secure :-1 s=READY pgs=6274 cs=0 l=1 rev1=1 rx=0x7ff99408e920 tx=0x7ff99407a1e0).stop 2021-03-09T10:27:50.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:50.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092cf0 unknown :-1 s=CLOSED pgs=6274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:50.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:50.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:50.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:50.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008bae0 con 0x7ff99c0adc20 2021-03-09T10:27:50.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026b70 con 0x7ff99c0613a0 2021-03-09T10:27:50.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088f40 con 0x7ff980083100 2021-03-09T10:27:50.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:50.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:50.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:50.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:50.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.378+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:50.388 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:50.382+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21087 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:50.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.382+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:50.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.382+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:50.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.382+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:50.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.382+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:50.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.382+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980039960 con 0x7ff980083100 2021-03-09T10:27:50.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.382+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092b60 con 0x7ff980083100 2021-03-09T10:27:50.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.382+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039470 secure :-1 s=READY pgs=6285 cs=0 l=1 rev1=1 rx=0x7ff980026af0 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:50.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff980083100 2021-03-09T10:27:50.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff980083100 2021-03-09T10:27:50.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980083100 2021-03-09T10:27:50.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.390+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:50.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.390+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:50.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:50.390+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:27:55.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:55.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:55.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039470 secure :-1 s=READY pgs=6285 cs=0 l=1 rev1=1 rx=0x7ff980026af0 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:55.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039470 secure :-1 s=READY pgs=6285 cs=0 l=1 rev1=1 rx=0x7ff980026af0 tx=0x7ff98c003e80).stop 2021-03-09T10:27:55.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:55.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039470 unknown :-1 s=CLOSED pgs=6285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:55.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:55.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:55.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:55.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053600 con 0x7ff99c0adc20 2021-03-09T10:27:55.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c10 con 0x7ff99c0613a0 2021-03-09T10:27:55.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004dc80 con 0x7ff9800301a0 2021-03-09T10:27:55.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:27:55.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:55.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:27:55.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:55.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:55.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:55.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:55.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:55.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:55.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026b70 con 0x7ff9800301a0 2021-03-09T10:27:55.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.382+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980039960 con 0x7ff9800301a0 2021-03-09T10:27:55.396 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:27:55.386+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21125 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:27:55.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.386+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092b10 secure :-1 s=READY pgs=6296 cs=0 l=1 rev1=1 rx=0x7ff98000a970 tx=0x7ff98c085320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:55.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff9800301a0 2021-03-09T10:27:55.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff9800301a0 2021-03-09T10:27:55.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.394+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c032a90 con 0x7ff9800301a0 2021-03-09T10:27:55.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.394+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:55.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.394+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:55.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:27:55.394+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:00.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:28:00.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:00.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092b10 secure :-1 s=READY pgs=6296 cs=0 l=1 rev1=1 rx=0x7ff98000a970 tx=0x7ff98c085320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:00.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092b10 secure :-1 s=READY pgs=6296 cs=0 l=1 rev1=1 rx=0x7ff98000a970 tx=0x7ff98c085320).stop 2021-03-09T10:28:00.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:00.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092b10 unknown :-1 s=CLOSED pgs=6296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:00.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:00.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:00.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:00.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a700 con 0x7ff99c0adc20 2021-03-09T10:28:00.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a840 con 0x7ff99c0613a0 2021-03-09T10:28:00.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039470 con 0x7ff980083100 2021-03-09T10:28:00.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:00.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:00.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:00.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:00.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:00.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:00.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:00.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:00.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:00.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800395b0 con 0x7ff980083100 2021-03-09T10:28:00.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.386+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017b40 con 0x7ff980083100 2021-03-09T10:28:00.398 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:00.390+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21161 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:00.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.390+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800053b0 secure :-1 s=READY pgs=6303 cs=0 l=1 rev1=1 rx=0x7ff994001300 tx=0x7ff994036960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:00.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940aa080 con 0x7ff980083100 2021-03-09T10:28:00.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980083100 2021-03-09T10:28:00.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.394+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99408dda0 con 0x7ff980083100 2021-03-09T10:28:00.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.394+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:00.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.398+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:00.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:00.398+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:02.182 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:28:02.185 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:28:05.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:05.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:05.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800053b0 secure :-1 s=READY pgs=6303 cs=0 l=1 rev1=1 rx=0x7ff994001300 tx=0x7ff994036960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:05.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800053b0 secure :-1 s=READY pgs=6303 cs=0 l=1 rev1=1 rx=0x7ff994001300 tx=0x7ff994036960).stop 2021-03-09T10:28:05.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800053b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:05.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800053b0 unknown :-1 s=CLOSED pgs=6303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:05.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:05.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:05.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:05.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003e440 con 0x7ff99c0adc20 2021-03-09T10:28:05.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0613a0 2021-03-09T10:28:05.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a700 con 0x7ff9800301a0 2021-03-09T10:28:05.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:05.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:05.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:05.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:05.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.390+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:05.400 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:05.390+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21199 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:05.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:05.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:05.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:05.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:05.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a840 con 0x7ff9800301a0 2021-03-09T10:28:05.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800395b0 con 0x7ff9800301a0 2021-03-09T10:28:05.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092a70 secure :-1 s=READY pgs=6317 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99405fe60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:05.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.398+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff9800301a0 2021-03-09T10:28:05.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.398+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994016370 con 0x7ff9800301a0 2021-03-09T10:28:05.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99400d1c0 con 0x7ff9800301a0 2021-03-09T10:28:05.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.402+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:05.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.402+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:05.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:05.402+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:10.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:10.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:10.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092a70 secure :-1 s=READY pgs=6317 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99405fe60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:10.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092a70 secure :-1 s=READY pgs=6317 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99405fe60).stop 2021-03-09T10:28:10.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:10.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092a70 unknown :-1 s=CLOSED pgs=6317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:10.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:10.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:10.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:10.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a700 con 0x7ff99c0adc20 2021-03-09T10:28:10.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800051e0 con 0x7ff99c0613a0 2021-03-09T10:28:10.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005320 con 0x7ff980083100 2021-03-09T10:28:10.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:10.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:10.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:10.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:10.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:10.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:10.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.398+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:10.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:10.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.398+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:10.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005490 con 0x7ff980083100 2021-03-09T10:28:10.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003a840 con 0x7ff980083100 2021-03-09T10:28:10.406 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:10.394+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21234 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:10.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.398+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039570 secure :-1 s=READY pgs=6327 cs=0 l=1 rev1=1 rx=0x7ff9940605a0 tx=0x7ff99400c270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:10.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980083100 2021-03-09T10:28:10.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980083100 2021-03-09T10:28:10.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.406+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:28:10.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.406+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:10.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.406+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:10.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:10.406+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:15.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:15.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:15.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039570 secure :-1 s=READY pgs=6327 cs=0 l=1 rev1=1 rx=0x7ff9940605a0 tx=0x7ff99400c270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:15.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039570 secure :-1 s=READY pgs=6327 cs=0 l=1 rev1=1 rx=0x7ff9940605a0 tx=0x7ff99400c270).stop 2021-03-09T10:28:15.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980039570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:15.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980039570 unknown :-1 s=CLOSED pgs=6327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:15.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:15.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:15.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:15.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005350 con 0x7ff99c0adc20 2021-03-09T10:28:15.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0613a0 2021-03-09T10:28:15.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a700 con 0x7ff9800301a0 2021-03-09T10:28:15.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:15.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:15.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:15.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:15.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:15.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:15.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:15.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:15.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:15.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98003a840 con 0x7ff9800301a0 2021-03-09T10:28:15.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.398+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005490 con 0x7ff9800301a0 2021-03-09T10:28:15.410 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:15.402+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21273 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:15.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092880 secure :-1 s=READY pgs=6336 cs=0 l=1 rev1=1 rx=0x7ff99400bfc0 tx=0x7ff994098f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:15.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994098ce0 con 0x7ff9800301a0 2021-03-09T10:28:15.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994020980 con 0x7ff9800301a0 2021-03-09T10:28:15.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.406+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff9800301a0 2021-03-09T10:28:15.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.406+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:15.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.410+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:15.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:15.410+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:20.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:20.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:20.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092880 secure :-1 s=READY pgs=6336 cs=0 l=1 rev1=1 rx=0x7ff99400bfc0 tx=0x7ff994098f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:20.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092880 secure :-1 s=READY pgs=6336 cs=0 l=1 rev1=1 rx=0x7ff99400bfc0 tx=0x7ff994098f30).stop 2021-03-09T10:28:20.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:20.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092880 unknown :-1 s=CLOSED pgs=6336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:20.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:20.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:20.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:20.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:20.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:20.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a700 con 0x7ff99c0adc20 2021-03-09T10:28:20.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f480 con 0x7ff99c0613a0 2021-03-09T10:28:20.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c360 con 0x7ff980083100 2021-03-09T10:28:20.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:20.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:20.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:20.412 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:20.406+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21308 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:20.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:20.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.406+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:20.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:20.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.406+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:20.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005500 con 0x7ff980083100 2021-03-09T10:28:20.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98003a840 con 0x7ff980083100 2021-03-09T10:28:20.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.406+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6350 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994075fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:20.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.410+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99406d1f0 con 0x7ff980083100 2021-03-09T10:28:20.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.410+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400edc0 con 0x7ff980083100 2021-03-09T10:28:20.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.414+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:28:20.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.414+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:20.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.414+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:20.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:20.414+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:25.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:25.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:25.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6350 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994075fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:25.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6350 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994075fa0).stop 2021-03-09T10:28:25.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:25.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:25.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:25.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:25.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800927d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:25.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f480 con 0x7ff99c0adc20 2021-03-09T10:28:25.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0613a0 2021-03-09T10:28:25.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092cc0 con 0x7ff9800301a0 2021-03-09T10:28:25.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:25.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:25.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:25.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:25.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800927d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:25.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:25.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:25.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:25.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:25.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092e00 con 0x7ff9800301a0 2021-03-09T10:28:25.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.406+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005500 con 0x7ff9800301a0 2021-03-09T10:28:25.419 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:25.410+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21347 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:25.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.410+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800927d0 secure :-1 s=READY pgs=6357 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994011400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:25.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.410+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400edc0 con 0x7ff9800301a0 2021-03-09T10:28:25.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.410+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098ce0 con 0x7ff9800301a0 2021-03-09T10:28:25.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.414+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940842c0 con 0x7ff9800301a0 2021-03-09T10:28:25.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.414+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:25.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.418+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:25.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:25.418+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:30.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800927d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:28:30.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800927d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:30.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800927d0 secure :-1 s=READY pgs=6357 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994011400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:30.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800927d0 secure :-1 s=READY pgs=6357 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994011400).stop 2021-03-09T10:28:30.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff9800927d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:30.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff9800927d0 unknown :-1 s=CLOSED pgs=6357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:30.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:30.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:30.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:30.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092cc0 con 0x7ff99c0adc20 2021-03-09T10:28:30.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008f840 con 0x7ff99c0613a0 2021-03-09T10:28:30.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005500 con 0x7ff980083100 2021-03-09T10:28:30.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:30.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:30.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:30.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:30.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:30.421 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:30.413+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21382 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:30.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:30.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:30.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:30.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:30.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800060c0 con 0x7ff980083100 2021-03-09T10:28:30.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092e00 con 0x7ff980083100 2021-03-09T10:28:30.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.413+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6370 cs=0 l=1 rev1=1 rx=0x7ff9940145d0 tx=0x7ff99400bef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:30.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994060230 con 0x7ff980083100 2021-03-09T10:28:30.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99406d420 con 0x7ff980083100 2021-03-09T10:28:30.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:28:30.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.421+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:30.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.421+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:30.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:30.421+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:32.230 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:28:32.233 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:28:35.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:35.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:35.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6370 cs=0 l=1 rev1=1 rx=0x7ff9940145d0 tx=0x7ff99400bef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:35.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6370 cs=0 l=1 rev1=1 rx=0x7ff9940145d0 tx=0x7ff99400bef0).stop 2021-03-09T10:28:35.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:35.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:35.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:35.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:35.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:35.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008f7f0 con 0x7ff99c0adc20 2021-03-09T10:28:35.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0613a0 2021-03-09T10:28:35.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005540 con 0x7ff9800301a0 2021-03-09T10:28:35.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:35.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:35.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:35.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:35.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:35.425 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:35.417+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21421 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:35.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:35.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:35.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:35.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:35.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092c20 con 0x7ff9800301a0 2021-03-09T10:28:35.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800060c0 con 0x7ff9800301a0 2021-03-09T10:28:35.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.417+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092730 secure :-1 s=READY pgs=6380 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff9940ac250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:35.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409da70 con 0x7ff9800301a0 2021-03-09T10:28:35.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940acdb0 con 0x7ff9800301a0 2021-03-09T10:28:35.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.425+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99402ada0 con 0x7ff9800301a0 2021-03-09T10:28:35.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.425+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:35.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.425+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:35.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:35.425+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:40.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:40.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:40.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092730 secure :-1 s=READY pgs=6380 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff9940ac250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:40.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092730 secure :-1 s=READY pgs=6380 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff9940ac250).stop 2021-03-09T10:28:40.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:40.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092730 unknown :-1 s=CLOSED pgs=6380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:40.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:40.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:40.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:40.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008f7f0 con 0x7ff99c0adc20 2021-03-09T10:28:40.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800060a0 con 0x7ff99c0613a0 2021-03-09T10:28:40.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800061e0 con 0x7ff980083100 2021-03-09T10:28:40.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:40.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:40.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:40.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:40.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:40.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:40.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:40.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:40.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:40.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092db0 con 0x7ff980083100 2021-03-09T10:28:40.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092c20 con 0x7ff980083100 2021-03-09T10:28:40.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6389 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99402a230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:40.431 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:40.421+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21456 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:40.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095ce0 con 0x7ff980083100 2021-03-09T10:28:40.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.421+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940acdb0 con 0x7ff980083100 2021-03-09T10:28:40.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.425+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99402ada0 con 0x7ff980083100 2021-03-09T10:28:40.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.425+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:40.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.429+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:40.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:40.429+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:45.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:45.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:45.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6389 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99402a230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:45.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6389 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99402a230).stop 2021-03-09T10:28:45.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:45.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:45.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:45.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:45.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:45.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980089110 con 0x7ff99c0adc20 2021-03-09T10:28:45.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005d90 con 0x7ff99c0613a0 2021-03-09T10:28:45.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008f7f0 con 0x7ff9800301a0 2021-03-09T10:28:45.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:45.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:45.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:45.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:45.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:45.433 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:45.425+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21495 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:45.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:45.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:45.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:45.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:45.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff9800301a0 2021-03-09T10:28:45.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092db0 con 0x7ff9800301a0 2021-03-09T10:28:45.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.425+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016ef0 secure :-1 s=READY pgs=6403 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff99400da90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:45.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.429+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940acdb0 con 0x7ff9800301a0 2021-03-09T10:28:45.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.429+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940acdb0 con 0x7ff9800301a0 2021-03-09T10:28:45.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.433+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99402ada0 con 0x7ff9800301a0 2021-03-09T10:28:45.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.433+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:45.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.433+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:45.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:45.433+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:50.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980016ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:50.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980016ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:50.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016ef0 secure :-1 s=READY pgs=6403 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff99400da90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:50.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016ef0 secure :-1 s=READY pgs=6403 cs=0 l=1 rev1=1 rx=0x7ff99408d060 tx=0x7ff99400da90).stop 2021-03-09T10:28:50.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980016ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:50.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016ef0 unknown :-1 s=CLOSED pgs=6403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:50.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:50.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:50.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:50.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980085700 con 0x7ff99c0adc20 2021-03-09T10:28:50.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014c00 con 0x7ff99c0613a0 2021-03-09T10:28:50.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008f7f0 con 0x7ff980083100 2021-03-09T10:28:50.435 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:50.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:50.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:50.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:50.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:50.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:50.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:50.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:50.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:50.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:28:50.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092db0 con 0x7ff980083100 2021-03-09T10:28:50.439 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:50.429+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21530 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:50.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.429+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6411 cs=0 l=1 rev1=1 rx=0x7ff994098d40 tx=0x7ff9940368f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:50.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.433+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405e1e0 con 0x7ff980083100 2021-03-09T10:28:50.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.437+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095ce0 con 0x7ff980083100 2021-03-09T10:28:50.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.437+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99402ada0 con 0x7ff980083100 2021-03-09T10:28:50.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.437+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:50.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.437+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:50.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:50.437+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:28:55.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:55.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:55.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6411 cs=0 l=1 rev1=1 rx=0x7ff994098d40 tx=0x7ff9940368f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:55.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6411 cs=0 l=1 rev1=1 rx=0x7ff994098d40 tx=0x7ff9940368f0).stop 2021-03-09T10:28:55.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:55.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:55.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:55.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:55.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:55.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0adc20 2021-03-09T10:28:55.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017370 con 0x7ff99c0613a0 2021-03-09T10:28:55.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008f4b0 con 0x7ff9800301a0 2021-03-09T10:28:55.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:28:55.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:55.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:28:55.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:55.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.433+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:55.735 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:28:55.437+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21569 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:28:55.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.437+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:55.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.437+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:55.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.437+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:55.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.437+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:55.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.437+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008f5f0 con 0x7ff9800301a0 2021-03-09T10:28:55.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.437+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026b70 con 0x7ff9800301a0 2021-03-09T10:28:55.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.437+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e80 secure :-1 s=READY pgs=6424 cs=0 l=1 rev1=1 rx=0x7ff98003a680 tx=0x7ff98c032da0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:55.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.441+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff9800301a0 2021-03-09T10:28:55.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.441+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff9800301a0 2021-03-09T10:28:55.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.445+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff9800301a0 2021-03-09T10:28:55.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.445+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:55.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.445+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:55.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:28:55.445+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:00.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980016e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:00.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980016e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:00.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e80 secure :-1 s=READY pgs=6424 cs=0 l=1 rev1=1 rx=0x7ff98003a680 tx=0x7ff98c032da0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:00.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e80 secure :-1 s=READY pgs=6424 cs=0 l=1 rev1=1 rx=0x7ff98003a680 tx=0x7ff98c032da0).stop 2021-03-09T10:29:00.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980016e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:00.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e80 unknown :-1 s=CLOSED pgs=6424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:00.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:00.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:00.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:00.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008f4b0 con 0x7ff99c0adc20 2021-03-09T10:29:00.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014a40 con 0x7ff99c0613a0 2021-03-09T10:29:00.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014b80 con 0x7ff980083100 2021-03-09T10:29:00.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:00.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:00.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:00.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:00.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.437+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:00.445 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:00.441+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21604 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:00.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.441+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:00.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.441+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:00.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.441+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:00.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.441+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:00.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.441+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800060a0 con 0x7ff980083100 2021-03-09T10:29:00.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.441+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092db0 con 0x7ff980083100 2021-03-09T10:29:00.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.441+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6435 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff99405e4e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:00.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.441+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:29:00.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.445+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940835b0 con 0x7ff980083100 2021-03-09T10:29:00.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.449+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:29:00.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.449+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:00.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.449+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:00.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:00.449+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:02.276 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:29:02.281 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:29:05.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:05.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:05.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6435 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff99405e4e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:05.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6435 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff99405e4e0).stop 2021-03-09T10:29:05.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:05.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:05.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:05.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:05.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:05.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037d30 con 0x7ff99c0adc20 2021-03-09T10:29:05.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0613a0 2021-03-09T10:29:05.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800172f0 con 0x7ff9800301a0 2021-03-09T10:29:05.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:05.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:05.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:05.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:05.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.441+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:05.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:05.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.445+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:05.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:05.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.445+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:05.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092d70 con 0x7ff9800301a0 2021-03-09T10:29:05.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800060a0 con 0x7ff9800301a0 2021-03-09T10:29:05.451 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:05.445+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21643 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:05.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.445+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e00 secure :-1 s=READY pgs=6441 cs=0 l=1 rev1=1 rx=0x7ff994016520 tx=0x7ff994086e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:05.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.449+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff9800301a0 2021-03-09T10:29:05.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.449+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff9800301a0 2021-03-09T10:29:05.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.449+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff9800301a0 2021-03-09T10:29:05.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.449+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:05.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.449+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:05.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:05.449+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:10.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980016e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:10.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980016e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:10.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e00 secure :-1 s=READY pgs=6441 cs=0 l=1 rev1=1 rx=0x7ff994016520 tx=0x7ff994086e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:10.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e00 secure :-1 s=READY pgs=6441 cs=0 l=1 rev1=1 rx=0x7ff994016520 tx=0x7ff994086e50).stop 2021-03-09T10:29:10.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980016e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:10.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980016e00 unknown :-1 s=CLOSED pgs=6441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:10.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:10.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:10.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:10.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e1c0 con 0x7ff99c0adc20 2021-03-09T10:29:10.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037ed0 con 0x7ff99c0613a0 2021-03-09T10:29:10.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053020 con 0x7ff980083100 2021-03-09T10:29:10.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:10.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:10.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:10.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:10.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:10.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:10.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:10.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:10.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:10.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980053160 con 0x7ff980083100 2021-03-09T10:29:10.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.445+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092d70 con 0x7ff980083100 2021-03-09T10:29:10.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.449+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6453 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff99408e2b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:10.456 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:10.449+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21677 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:10.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.449+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:29:10.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.449+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407eed0 con 0x7ff980083100 2021-03-09T10:29:10.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.453+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001cd0 con 0x7ff980083100 2021-03-09T10:29:10.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.453+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:10.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.453+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:10.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:10.457+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:15.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:15.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:15.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6453 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff99408e2b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:15.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6453 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff99408e2b0).stop 2021-03-09T10:29:15.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:15.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:15.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:15.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:15.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:15.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053020 con 0x7ff99c0adc20 2021-03-09T10:29:15.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800096e0 con 0x7ff99c0613a0 2021-03-09T10:29:15.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800460a0 con 0x7ff9800301a0 2021-03-09T10:29:15.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:15.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:15.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:15.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:15.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.449+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:15.458 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:15.453+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21717 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:15.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.453+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:15.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.453+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:15.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.453+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:15.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.453+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:15.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.453+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff9800301a0 2021-03-09T10:29:15.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.453+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980053160 con 0x7ff9800301a0 2021-03-09T10:29:15.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.457+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d70 secure :-1 s=READY pgs=6468 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99400eee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:15.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.461+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff9800301a0 2021-03-09T10:29:15.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.465+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407eed0 con 0x7ff9800301a0 2021-03-09T10:29:15.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.465+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001cd0 con 0x7ff9800301a0 2021-03-09T10:29:15.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.465+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:15.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.465+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:15.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:15.465+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:20.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:20.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:20.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d70 secure :-1 s=READY pgs=6468 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99400eee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:20.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d70 secure :-1 s=READY pgs=6468 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99400eee0).stop 2021-03-09T10:29:20.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980092d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:20.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980092d70 unknown :-1 s=CLOSED pgs=6468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:20.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:20.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:20.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:20.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053020 con 0x7ff99c0adc20 2021-03-09T10:29:20.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046240 con 0x7ff99c0613a0 2021-03-09T10:29:20.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e1c0 con 0x7ff980083100 2021-03-09T10:29:20.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:20.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:20.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:20.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:20.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.453+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:20.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.457+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:20.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.457+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:20.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.457+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:20.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.457+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:20.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.457+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:29:20.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.457+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008f5f0 con 0x7ff980083100 2021-03-09T10:29:20.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.457+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6473 cs=0 l=1 rev1=1 rx=0x7ff98c0a9930 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:20.465 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:20.457+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21752 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:20.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.461+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:29:20.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.461+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079fc0 con 0x7ff980083100 2021-03-09T10:29:20.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.465+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:29:20.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.465+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:20.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.465+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:20.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:20.465+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:25.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:25.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:25.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6473 cs=0 l=1 rev1=1 rx=0x7ff98c0a9930 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:25.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6473 cs=0 l=1 rev1=1 rx=0x7ff98c0a9930 tx=0x7ff98c003e80).stop 2021-03-09T10:29:25.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:25.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:25.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:25.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:25.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:25.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053020 con 0x7ff99c0adc20 2021-03-09T10:29:25.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800463e0 con 0x7ff99c0613a0 2021-03-09T10:29:25.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e1c0 con 0x7ff9800301a0 2021-03-09T10:29:25.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:25.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:25.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:25.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:25.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.457+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:25.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.461+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:25.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.461+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:25.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.461+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:25.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.461+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:25.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.461+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff9800301a0 2021-03-09T10:29:25.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.461+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980053160 con 0x7ff9800301a0 2021-03-09T10:29:25.468 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:25.461+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21791 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:25.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.461+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 secure :-1 s=READY pgs=6486 cs=0 l=1 rev1=1 rx=0x7ff994016520 tx=0x7ff99402a1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:25.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.465+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994034000 con 0x7ff9800301a0 2021-03-09T10:29:25.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.465+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994068130 con 0x7ff9800301a0 2021-03-09T10:29:25.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.469+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff9800301a0 2021-03-09T10:29:25.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.469+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:25.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.469+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:25.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:25.469+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:30.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980093470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:30.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980093470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:30.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 secure :-1 s=READY pgs=6486 cs=0 l=1 rev1=1 rx=0x7ff994016520 tx=0x7ff99402a1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:30.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 secure :-1 s=READY pgs=6486 cs=0 l=1 rev1=1 rx=0x7ff994016520 tx=0x7ff99402a1f0).stop 2021-03-09T10:29:30.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980093470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:30.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 unknown :-1 s=CLOSED pgs=6486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:30.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:30.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:30.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:30.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053020 con 0x7ff99c0adc20 2021-03-09T10:29:30.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c8d0 con 0x7ff99c0613a0 2021-03-09T10:29:30.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e1c0 con 0x7ff980083100 2021-03-09T10:29:30.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:30.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:30.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:30.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:30.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.461+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:30.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.465+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:30.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.465+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:30.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.465+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:30.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.465+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:30.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.465+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:29:30.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.465+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980053160 con 0x7ff980083100 2021-03-09T10:29:30.473 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:30.465+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21826 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:30.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.465+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6496 cs=0 l=1 rev1=1 rx=0x7ff9940682f0 tx=0x7ff994034070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:30.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.469+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:29:30.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.469+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:29:30.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.473+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700b0 con 0x7ff980083100 2021-03-09T10:29:30.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.473+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:30.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.473+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:30.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:30.473+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:32.338 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:29:32.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:29:35.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:35.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:35.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6496 cs=0 l=1 rev1=1 rx=0x7ff9940682f0 tx=0x7ff994034070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:35.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6496 cs=0 l=1 rev1=1 rx=0x7ff9940682f0 tx=0x7ff994034070).stop 2021-03-09T10:29:35.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:35.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:35.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:35.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:35.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:35.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053020 con 0x7ff99c0adc20 2021-03-09T10:29:35.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d680 con 0x7ff99c0613a0 2021-03-09T10:29:35.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.465+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ca70 con 0x7ff9800301a0 2021-03-09T10:29:35.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:35.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:35.474 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:35.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:35.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:35.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:35.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:35.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:35.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:35.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008e1c0 con 0x7ff9800301a0 2021-03-09T10:29:35.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980053160 con 0x7ff9800301a0 2021-03-09T10:29:35.477 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:35.469+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21865 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:35.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.469+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 secure :-1 s=READY pgs=6505 cs=0 l=1 rev1=1 rx=0x7ff994095ff0 tx=0x7ff9940604d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:35.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.473+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff9800301a0 2021-03-09T10:29:35.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.473+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff9800301a0 2021-03-09T10:29:35.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.477+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940682f0 con 0x7ff9800301a0 2021-03-09T10:29:35.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.477+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:35.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.481+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:35.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:35.481+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:40.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980093470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:40.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980093470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:40.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 secure :-1 s=READY pgs=6505 cs=0 l=1 rev1=1 rx=0x7ff994095ff0 tx=0x7ff9940604d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:40.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 secure :-1 s=READY pgs=6505 cs=0 l=1 rev1=1 rx=0x7ff994095ff0 tx=0x7ff9940604d0).stop 2021-03-09T10:29:40.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 msgr2=0x7ff980093470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:40.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff9800301a0 0x7ff980093470 unknown :-1 s=CLOSED pgs=6505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:40.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:40.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:40.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:40.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008eb70 con 0x7ff99c0adc20 2021-03-09T10:29:40.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d680 con 0x7ff99c0613a0 2021-03-09T10:29:40.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.469+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a670 con 0x7ff980083100 2021-03-09T10:29:40.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:40.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:40.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:40.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:40.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:40.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:40.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:40.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:40.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:40.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008ddd0 con 0x7ff980083100 2021-03-09T10:29:40.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.473+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008f5f0 con 0x7ff980083100 2021-03-09T10:29:40.482 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:40.473+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21900 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:40.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.477+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6518 cs=0 l=1 rev1=1 rx=0x7ff98c07a280 tx=0x7ff98c079770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:40.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.477+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:29:40.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.477+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c022130 con 0x7ff980083100 2021-03-09T10:29:40.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.481+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:29:40.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.481+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:40.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.485+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:40.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:40.485+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:45.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:45.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:45.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6518 cs=0 l=1 rev1=1 rx=0x7ff98c07a280 tx=0x7ff98c079770).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:45.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6518 cs=0 l=1 rev1=1 rx=0x7ff98c07a280 tx=0x7ff98c079770).stop 2021-03-09T10:29:45.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:45.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:45.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:45.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:45.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:45.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026240 con 0x7ff99c0adc20 2021-03-09T10:29:45.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800304e0 con 0x7ff99c0613a0 2021-03-09T10:29:45.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:45.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d680 con 0x7ff98000ac80 2021-03-09T10:29:45.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.472+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:45.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.476+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:45.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.476+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:45.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.476+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:45.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.476+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:45.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.476+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:45.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.476+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:45.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.476+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:45.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.476+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001a650 con 0x7ff98000ac80 2021-03-09T10:29:45.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.476+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008e1c0 con 0x7ff98000ac80 2021-03-09T10:29:45.485 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:45.476+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21939 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:45.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.476+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 secure :-1 s=READY pgs=6527 cs=0 l=1 rev1=1 rx=0x7ff994020730 tx=0x7ff994001f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:45.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.480+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff98000ac80 2021-03-09T10:29:45.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.480+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095120 con 0x7ff98000ac80 2021-03-09T10:29:45.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.484+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994029250 con 0x7ff98000ac80 2021-03-09T10:29:45.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.484+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:45.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.484+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:45.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:45.484+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:50.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 msgr2=0x7ff980093470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:50.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 msgr2=0x7ff980093470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:50.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 secure :-1 s=READY pgs=6527 cs=0 l=1 rev1=1 rx=0x7ff994020730 tx=0x7ff994001f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:50.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 secure :-1 s=READY pgs=6527 cs=0 l=1 rev1=1 rx=0x7ff994020730 tx=0x7ff994001f40).stop 2021-03-09T10:29:50.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 msgr2=0x7ff980093470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:50.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 unknown :-1 s=CLOSED pgs=6527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:50.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:50.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:50.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:50.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093b00 con 0x7ff99c0adc20 2021-03-09T10:29:50.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026220 con 0x7ff99c0613a0 2021-03-09T10:29:50.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d680 con 0x7ff980083100 2021-03-09T10:29:50.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:50.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:50.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:50.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:50.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.480+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:50.487 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:50.480+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21974 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:50.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.484+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:50.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.484+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:50.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.484+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:50.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.484+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:50.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.484+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800093a0 con 0x7ff980083100 2021-03-09T10:29:50.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.484+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001a650 con 0x7ff980083100 2021-03-09T10:29:50.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.484+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6539 cs=0 l=1 rev1=1 rx=0x7ff9940ae370 tx=0x7ff99405f1c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:50.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.484+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994068130 con 0x7ff980083100 2021-03-09T10:29:50.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.484+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095350 con 0x7ff980083100 2021-03-09T10:29:50.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.492+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994029250 con 0x7ff980083100 2021-03-09T10:29:50.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.492+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:50.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.492+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:50.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:50.492+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:29:55.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.480+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:55.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.480+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:55.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.480+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6539 cs=0 l=1 rev1=1 rx=0x7ff9940ae370 tx=0x7ff99405f1c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:55.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.480+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6539 cs=0 l=1 rev1=1 rx=0x7ff9940ae370 tx=0x7ff99405f1c0).stop 2021-03-09T10:29:55.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:55.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:55.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:55.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:55.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:55.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026220 con 0x7ff99c0adc20 2021-03-09T10:29:55.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d680 con 0x7ff99c0613a0 2021-03-09T10:29:55.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037ed0 con 0x7ff98000ac80 2021-03-09T10:29:55.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:29:55.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:55.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:29:55.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:55.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:55.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:55.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:55.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:55.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:55.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff98000ac80 2021-03-09T10:29:55.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.484+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800093a0 con 0x7ff98000ac80 2021-03-09T10:29:55.493 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:29:55.488+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22013 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:29:55.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.488+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 secure :-1 s=READY pgs=6553 cs=0 l=1 rev1=1 rx=0x7ff99409d620 tx=0x7ff9940ae070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:55.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.488+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409d280 con 0x7ff98000ac80 2021-03-09T10:29:55.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.492+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095120 con 0x7ff98000ac80 2021-03-09T10:29:55.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.496+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994029250 con 0x7ff98000ac80 2021-03-09T10:29:55.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.496+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:55.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.496+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:55.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:29:55.496+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:00.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 msgr2=0x7ff980093470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:00.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 msgr2=0x7ff980093470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:00.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 secure :-1 s=READY pgs=6553 cs=0 l=1 rev1=1 rx=0x7ff99409d620 tx=0x7ff9940ae070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:00.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 secure :-1 s=READY pgs=6553 cs=0 l=1 rev1=1 rx=0x7ff99409d620 tx=0x7ff9940ae070).stop 2021-03-09T10:30:00.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 msgr2=0x7ff980093470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:00.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac80 0x7ff980093470 unknown :-1 s=CLOSED pgs=6553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:00.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:00.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:00.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:00.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026220 con 0x7ff99c0adc20 2021-03-09T10:30:00.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d680 con 0x7ff99c0613a0 2021-03-09T10:30:00.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037ed0 con 0x7ff980083100 2021-03-09T10:30:00.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:00.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:00.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:00.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:00.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.488+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:00.497 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:00.492+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22049 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:00.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.492+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:00.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.492+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:00.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.492+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:00.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.492+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:00.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.492+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:30:00.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.492+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800093a0 con 0x7ff980083100 2021-03-09T10:30:00.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.492+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6562 cs=0 l=1 rev1=1 rx=0x7ff99407e5d0 tx=0x7ff99405eeb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:00.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.492+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994094540 con 0x7ff980083100 2021-03-09T10:30:00.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.496+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095120 con 0x7ff980083100 2021-03-09T10:30:00.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.500+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99409b560 con 0x7ff980083100 2021-03-09T10:30:00.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.500+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:00.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.500+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:00.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:00.500+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:02.394 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:30:02.397 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:30:05.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:05.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:05.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6562 cs=0 l=1 rev1=1 rx=0x7ff99407e5d0 tx=0x7ff99405eeb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:05.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6562 cs=0 l=1 rev1=1 rx=0x7ff99407e5d0 tx=0x7ff99405eeb0).stop 2021-03-09T10:30:05.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:05.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:05.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:05.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:05.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:05.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026220 con 0x7ff99c0adc20 2021-03-09T10:30:05.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039950 con 0x7ff99c0613a0 2021-03-09T10:30:05.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009a10 con 0x7ff98000ac10 2021-03-09T10:30:05.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:05.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:05.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:05.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:05.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.492+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:05.502 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:05.496+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22086 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:05.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.496+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:05.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.496+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:05.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.496+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:05.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.496+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:05.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.496+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff98000ac10 2021-03-09T10:30:05.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.496+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ddd0 con 0x7ff98000ac10 2021-03-09T10:30:05.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.500+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c280 secure :-1 s=READY pgs=6573 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c004f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:05.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.500+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:30:05.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.504+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:30:05.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.504+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:30:05.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.504+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:05.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.508+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:05.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:05.508+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:10.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:10.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:10.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c280 secure :-1 s=READY pgs=6573 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c004f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:10.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c280 secure :-1 s=READY pgs=6573 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c004f20).stop 2021-03-09T10:30:10.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:10.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c280 unknown :-1 s=CLOSED pgs=6573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:10.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:10.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:10.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:10.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a7b0 con 0x7ff99c0adc20 2021-03-09T10:30:10.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026220 con 0x7ff99c0613a0 2021-03-09T10:30:10.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009a10 con 0x7ff980083100 2021-03-09T10:30:10.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:10.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:10.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:10.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.496+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:10.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.500+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:10.506 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:10.500+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22122 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:10.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.500+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:10.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.500+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:10.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.500+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:10.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.500+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:10.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.500+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:30:10.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.500+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800093a0 con 0x7ff980083100 2021-03-09T10:30:10.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.504+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6585 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994084f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:10.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.504+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405e9a0 con 0x7ff980083100 2021-03-09T10:30:10.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.508+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405e9a0 con 0x7ff980083100 2021-03-09T10:30:10.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.508+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:30:10.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.508+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:10.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.512+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:10.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:10.512+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:15.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:15.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:15.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6585 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994084f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:15.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6585 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994084f90).stop 2021-03-09T10:30:15.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:15.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:15.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:15.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:15.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:15.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a7b0 con 0x7ff99c0adc20 2021-03-09T10:30:15.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002f850 con 0x7ff99c0613a0 2021-03-09T10:30:15.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026220 con 0x7ff98000ac10 2021-03-09T10:30:15.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:15.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:15.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:15.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:15.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.500+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:15.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.504+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:15.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.504+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:15.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.504+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:15.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.504+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:15.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.504+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff98000ac10 2021-03-09T10:30:15.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.504+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800093a0 con 0x7ff98000ac10 2021-03-09T10:30:15.512 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:15.504+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22160 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:15.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.504+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 secure :-1 s=READY pgs=6591 cs=0 l=1 rev1=1 rx=0x7ff9940014e0 tx=0x7ff994001400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:15.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.508+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff98000ac10 2021-03-09T10:30:15.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.508+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff98000ac10 2021-03-09T10:30:15.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.512+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:30:15.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.512+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:15.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.512+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:15.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:15.512+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:20.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:20.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:20.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 secure :-1 s=READY pgs=6591 cs=0 l=1 rev1=1 rx=0x7ff9940014e0 tx=0x7ff994001400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:20.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 secure :-1 s=READY pgs=6591 cs=0 l=1 rev1=1 rx=0x7ff9940014e0 tx=0x7ff994001400).stop 2021-03-09T10:30:20.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:20.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 unknown :-1 s=CLOSED pgs=6591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:20.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:20.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:20.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:20.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a7b0 con 0x7ff99c0adc20 2021-03-09T10:30:20.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026220 con 0x7ff99c0613a0 2021-03-09T10:30:20.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009a10 con 0x7ff980083100 2021-03-09T10:30:20.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:20.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:20.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:20.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.504+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:20.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.508+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:20.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.508+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:20.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.508+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:20.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.508+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:20.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.508+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:20.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.508+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:30:20.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.508+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800093a0 con 0x7ff980083100 2021-03-09T10:30:20.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.508+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6601 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994014190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:20.516 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:20.508+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22197 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:20.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.508+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:30:20.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.508+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:30:20.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.512+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:30:20.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.512+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:20.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.516+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:20.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:20.516+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:25.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:25.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:25.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6601 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994014190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:25.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6601 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994014190).stop 2021-03-09T10:30:25.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:25.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:25.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:25.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:25.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:25.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a7b0 con 0x7ff99c0adc20 2021-03-09T10:30:25.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009a10 con 0x7ff99c0613a0 2021-03-09T10:30:25.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037ed0 con 0x7ff98000ac10 2021-03-09T10:30:25.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:25.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:25.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:25.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.508+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:25.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.512+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:25.519 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:25.512+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22231 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:25.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.516+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:25.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.516+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:25.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.516+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:25.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.516+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:25.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.516+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff98000ac10 2021-03-09T10:30:25.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.516+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ddd0 con 0x7ff98000ac10 2021-03-09T10:30:25.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.516+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 secure :-1 s=READY pgs=6619 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c07b020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:25.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.520+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:30:25.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.520+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:30:25.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.524+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:30:25.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.524+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:25.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.524+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:25.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:25.524+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:30.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:30:30.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:30.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 secure :-1 s=READY pgs=6619 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c07b020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:30.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 secure :-1 s=READY pgs=6619 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c07b020).stop 2021-03-09T10:30:30.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:30.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 unknown :-1 s=CLOSED pgs=6619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:30.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:30.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:30.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:30.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a7b0 con 0x7ff99c0adc20 2021-03-09T10:30:30.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092bf0 con 0x7ff99c0613a0 2021-03-09T10:30:30.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009a10 con 0x7ff980083100 2021-03-09T10:30:30.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:30.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:30.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:30.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:30.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:30.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:30.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:30.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:30.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:30.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:30:30.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800093a0 con 0x7ff980083100 2021-03-09T10:30:30.525 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:30.516+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22271 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:30.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.516+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6623 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400c150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:30.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.520+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069fd0 con 0x7ff980083100 2021-03-09T10:30:30.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.520+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069fd0 con 0x7ff980083100 2021-03-09T10:30:30.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.524+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:30:30.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.524+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:30.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.524+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:30.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:30.524+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:32.446 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:30:32.449 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:30:35.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:35.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:35.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6623 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400c150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:35.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 secure :-1 s=READY pgs=6623 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400c150).stop 2021-03-09T10:30:35.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980005010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:35.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980005010 unknown :-1 s=CLOSED pgs=6623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:35.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:35.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:35.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:35.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a7b0 con 0x7ff99c0adc20 2021-03-09T10:30:35.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092bf0 con 0x7ff99c0613a0 2021-03-09T10:30:35.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c720 con 0x7ff98000ac10 2021-03-09T10:30:35.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:35.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:35.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:35.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:35.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:35.528 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:35.520+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22305 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:35.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:35.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:35.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:35.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:35.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff98000ac10 2021-03-09T10:30:35.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.520+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800093a0 con 0x7ff98000ac10 2021-03-09T10:30:35.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.524+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 secure :-1 s=READY pgs=6635 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99400d230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:35.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.524+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:30:35.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.528+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:30:35.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.532+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff98000ac10 2021-03-09T10:30:35.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.532+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:35.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.532+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:35.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:35.532+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:40.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:30:40.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:40.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 secure :-1 s=READY pgs=6635 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99400d230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:40.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 secure :-1 s=READY pgs=6635 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99400d230).stop 2021-03-09T10:30:40.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:40.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c230 unknown :-1 s=CLOSED pgs=6635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:40.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:40.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:40.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:40.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a7b0 con 0x7ff99c0adc20 2021-03-09T10:30:40.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043da0 con 0x7ff99c0613a0 2021-03-09T10:30:40.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092bf0 con 0x7ff980083100 2021-03-09T10:30:40.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:40.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:40.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:40.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:40.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:40.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:40.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:40.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:40.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:40.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:30:40.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ddd0 con 0x7ff980083100 2021-03-09T10:30:40.533 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:40.524+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22345 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:40.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.524+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093fa0 secure :-1 s=READY pgs=6647 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c08f260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:40.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.528+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c07dcb0 con 0x7ff980083100 2021-03-09T10:30:40.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.528+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c09fa80 con 0x7ff980083100 2021-03-09T10:30:40.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.536+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:30:40.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.536+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:40.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.536+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:40.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:40.536+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:45.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:45.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:45.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093fa0 secure :-1 s=READY pgs=6647 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c08f260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:45.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093fa0 secure :-1 s=READY pgs=6647 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c08f260).stop 2021-03-09T10:30:45.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:45.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093fa0 unknown :-1 s=CLOSED pgs=6647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:45.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:45.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:45.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:45.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094490 con 0x7ff99c0adc20 2021-03-09T10:30:45.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980043da0 con 0x7ff99c0613a0 2021-03-09T10:30:45.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092d90 con 0x7ff98000ac10 2021-03-09T10:30:45.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:45.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:45.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:45.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:45.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:45.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:45.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:45.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:45.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:45.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff98000ac10 2021-03-09T10:30:45.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.528+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800093a0 con 0x7ff98000ac10 2021-03-09T10:30:45.537 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:45.528+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22379 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:45.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.532+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093400 secure :-1 s=READY pgs=6657 cs=0 l=1 rev1=1 rx=0x7ff980054b10 tx=0x7ff994036960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:45.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.536+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:30:45.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.536+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:30:45.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.540+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:30:45.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.540+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:45.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.540+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:45.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:45.540+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:50.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980093400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:30:50.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980093400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:50.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093400 secure :-1 s=READY pgs=6657 cs=0 l=1 rev1=1 rx=0x7ff980054b10 tx=0x7ff994036960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:50.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093400 secure :-1 s=READY pgs=6657 cs=0 l=1 rev1=1 rx=0x7ff980054b10 tx=0x7ff994036960).stop 2021-03-09T10:30:50.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980093400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:50.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093400 unknown :-1 s=CLOSED pgs=6657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:50.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:50.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:50.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:50.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092d90 con 0x7ff99c0adc20 2021-03-09T10:30:50.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0613a0 2021-03-09T10:30:50.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094410 con 0x7ff980083100 2021-03-09T10:30:50.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:50.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:50.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:50.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:50.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:50.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:50.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:50.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:50.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:50.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000c230 con 0x7ff980083100 2021-03-09T10:30:50.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ddd0 con 0x7ff980083100 2021-03-09T10:30:50.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.532+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6665 cs=0 l=1 rev1=1 rx=0x7ff98c009ad0 tx=0x7ff98c08f1a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:50.542 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:50.532+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22419 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:50.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.536+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c07dcb0 con 0x7ff980083100 2021-03-09T10:30:50.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.536+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:30:50.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.540+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:30:50.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.540+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:50.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.540+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:50.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:50.540+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:30:55.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:55.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:55.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6665 cs=0 l=1 rev1=1 rx=0x7ff98c009ad0 tx=0x7ff98c08f1a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:55.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6665 cs=0 l=1 rev1=1 rx=0x7ff98c009ad0 tx=0x7ff98c08f1a0).stop 2021-03-09T10:30:55.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:55.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=CLOSED pgs=6665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:55.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:55.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:55.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:55.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094410 con 0x7ff99c0adc20 2021-03-09T10:30:55.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c8b0 con 0x7ff99c0613a0 2021-03-09T10:30:55.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff98000ac10 2021-03-09T10:30:55.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:30:55.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:30:55.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:55.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:55.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:55.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:55.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:55.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:55.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:55.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980085630 con 0x7ff98000ac10 2021-03-09T10:30:55.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800093a0 con 0x7ff98000ac10 2021-03-09T10:30:55.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.536+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 secure :-1 s=READY pgs=6677 cs=0 l=1 rev1=1 rx=0x7ff98004f610 tx=0x7ff99402a5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:55.546 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:30:55.536+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22453 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:30:55.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.540+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:30:55.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.540+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff98000ac10 2021-03-09T10:30:55.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.544+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:30:55.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.544+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:55.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.544+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:55.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:30:55.544+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:00.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:00.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:00.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 secure :-1 s=READY pgs=6677 cs=0 l=1 rev1=1 rx=0x7ff98004f610 tx=0x7ff99402a5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:00.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 secure :-1 s=READY pgs=6677 cs=0 l=1 rev1=1 rx=0x7ff98004f610 tx=0x7ff99402a5d0).stop 2021-03-09T10:31:00.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:00.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 unknown :-1 s=CLOSED pgs=6677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:00.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:00.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:00.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:00.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0adc20 2021-03-09T10:31:00.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008bb30 con 0x7ff99c0613a0 2021-03-09T10:31:00.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016fb0 con 0x7ff980083100 2021-03-09T10:31:00.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:00.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:00.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:00.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:00.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:00.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:00.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:00.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:00.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:00.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094410 con 0x7ff980083100 2021-03-09T10:31:00.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980085630 con 0x7ff980083100 2021-03-09T10:31:00.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.539+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6687 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99405f460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:00.550 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:00.543+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22493 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:00.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.543+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009a20 con 0x7ff980083100 2021-03-09T10:31:00.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.543+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940700e0 con 0x7ff980083100 2021-03-09T10:31:00.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.547+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994084d80 con 0x7ff980083100 2021-03-09T10:31:00.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.547+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:00.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.551+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:00.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:00.551+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:02.488 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:31:02.491 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:31:05.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:05.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:05.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6687 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99405f460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:05.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6687 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99405f460).stop 2021-03-09T10:31:05.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:05.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=CLOSED pgs=6687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:05.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:05.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:05.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:05.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016fb0 con 0x7ff99c0adc20 2021-03-09T10:31:05.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c8b0 con 0x7ff99c0613a0 2021-03-09T10:31:05.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff98000ac10 2021-03-09T10:31:05.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:05.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:05.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:05.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:05.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:05.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:05.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:05.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:05.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:05.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980085630 con 0x7ff98000ac10 2021-03-09T10:31:05.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.543+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094410 con 0x7ff98000ac10 2021-03-09T10:31:05.554 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:05.547+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22527 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:05.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.547+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 secure :-1 s=READY pgs=6698 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff9940099c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:05.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.547+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff98000ac10 2021-03-09T10:31:05.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.547+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994016370 con 0x7ff98000ac10 2021-03-09T10:31:05.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.551+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff98000ac10 2021-03-09T10:31:05.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.551+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:05.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.551+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:05.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:05.551+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:10.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:10.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:10.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 secure :-1 s=READY pgs=6698 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff9940099c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:10.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 secure :-1 s=READY pgs=6698 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff9940099c0).stop 2021-03-09T10:31:10.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000c3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:10.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000c3c0 unknown :-1 s=CLOSED pgs=6698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:10.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:10.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:10.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:10.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:10.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:10.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0adc20 2021-03-09T10:31:10.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016fb0 con 0x7ff99c0613a0 2021-03-09T10:31:10.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094410 con 0x7ff980083100 2021-03-09T10:31:10.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:10.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:10.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:10.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:10.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:10.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:10.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:10.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093400 con 0x7ff980083100 2021-03-09T10:31:10.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000c230 con 0x7ff980083100 2021-03-09T10:31:10.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.547+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6709 cs=0 l=1 rev1=1 rx=0x7ff980003e70 tx=0x7ff98c006650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:10.560 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:10.551+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22567 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:10.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.551+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:31:10.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.551+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980083100 2021-03-09T10:31:10.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.555+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:31:10.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.555+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:10.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.555+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:10.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:10.555+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:15.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:15.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:15.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6709 cs=0 l=1 rev1=1 rx=0x7ff980003e70 tx=0x7ff98c006650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:15.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6709 cs=0 l=1 rev1=1 rx=0x7ff980003e70 tx=0x7ff98c006650).stop 2021-03-09T10:31:15.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:15.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=CLOSED pgs=6709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:15.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:15.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:15.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:15.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0adc20 2021-03-09T10:31:15.558 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f640 con 0x7ff99c0613a0 2021-03-09T10:31:15.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016fb0 con 0x7ff98000ac10 2021-03-09T10:31:15.559 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:15.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:15.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:15.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:15.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.551+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:15.561 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:15.555+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22601 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:15.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:15.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.555+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:15.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:15.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.555+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:15.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000c230 con 0x7ff98000ac10 2021-03-09T10:31:15.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980085630 con 0x7ff98000ac10 2021-03-09T10:31:15.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.555+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 secure :-1 s=READY pgs=6724 cs=0 l=1 rev1=1 rx=0x7ff9940aede0 tx=0x7ff99405f510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:15.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.559+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994068130 con 0x7ff98000ac10 2021-03-09T10:31:15.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.563+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994068130 con 0x7ff98000ac10 2021-03-09T10:31:15.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.563+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700b0 con 0x7ff98000ac10 2021-03-09T10:31:15.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.563+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:15.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.563+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:15.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:15.563+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:20.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980093590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:20.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980093590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:20.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 secure :-1 s=READY pgs=6724 cs=0 l=1 rev1=1 rx=0x7ff9940aede0 tx=0x7ff99405f510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:20.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 secure :-1 s=READY pgs=6724 cs=0 l=1 rev1=1 rx=0x7ff9940aede0 tx=0x7ff99405f510).stop 2021-03-09T10:31:20.561 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980093590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:20.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 unknown :-1 s=CLOSED pgs=6724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:20.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:20.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:20.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:20.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026cc0 con 0x7ff99c0adc20 2021-03-09T10:31:20.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094410 con 0x7ff99c0613a0 2021-03-09T10:31:20.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:31:20.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:20.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:20.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:20.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:20.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:20.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:20.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:20.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:20.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:20.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980085630 con 0x7ff980083100 2021-03-09T10:31:20.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.555+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000c230 con 0x7ff980083100 2021-03-09T10:31:20.567 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:20.559+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22641 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:20.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.559+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6729 cs=0 l=1 rev1=1 rx=0x7ff994009a70 tx=0x7ff994068130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:20.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.559+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994012df0 con 0x7ff980083100 2021-03-09T10:31:20.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.559+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940700b0 con 0x7ff980083100 2021-03-09T10:31:20.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.563+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700b0 con 0x7ff980083100 2021-03-09T10:31:20.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.563+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:20.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.563+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:20.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:20.563+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:25.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:25.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:25.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6729 cs=0 l=1 rev1=1 rx=0x7ff994009a70 tx=0x7ff994068130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:25.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6729 cs=0 l=1 rev1=1 rx=0x7ff994009a70 tx=0x7ff994068130).stop 2021-03-09T10:31:25.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:25.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=CLOSED pgs=6729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:25.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:25.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:25.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:25.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff99c0adc20 2021-03-09T10:31:25.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800093a0 con 0x7ff99c0613a0 2021-03-09T10:31:25.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800094e0 con 0x7ff98000ac10 2021-03-09T10:31:25.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:25.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:25.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:25.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:25.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:25.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:25.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:25.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:25.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:25.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000c170 con 0x7ff98000ac10 2021-03-09T10:31:25.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.559+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980085630 con 0x7ff98000ac10 2021-03-09T10:31:25.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.563+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 secure :-1 s=READY pgs=6741 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99407e370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:25.572 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:25.563+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22675 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:25.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.563+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400d5d0 con 0x7ff98000ac10 2021-03-09T10:31:25.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.563+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff98000ac10 2021-03-09T10:31:25.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.567+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700b0 con 0x7ff98000ac10 2021-03-09T10:31:25.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.567+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:25.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.571+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:25.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:25.571+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:30.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980093590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:30.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980093590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:30.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 secure :-1 s=READY pgs=6741 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99407e370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:30.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 secure :-1 s=READY pgs=6741 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99407e370).stop 2021-03-09T10:31:30.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980093590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:30.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980093590 unknown :-1 s=CLOSED pgs=6741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:30.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:30.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:30.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:30.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017270 con 0x7ff99c0adc20 2021-03-09T10:31:30.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094410 con 0x7ff99c0613a0 2021-03-09T10:31:30.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026af0 con 0x7ff980083100 2021-03-09T10:31:30.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:30.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:30.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:30.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.563+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:30.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.567+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:30.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.567+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:30.574 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:30.567+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22715 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:30.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.567+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:30.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.567+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:30.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.567+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:30.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.567+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026c30 con 0x7ff980083100 2021-03-09T10:31:30.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.567+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000c170 con 0x7ff980083100 2021-03-09T10:31:30.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.567+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6753 cs=0 l=1 rev1=1 rx=0x7ff99409bb80 tx=0x7ff9940948a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:30.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.567+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994060230 con 0x7ff980083100 2021-03-09T10:31:30.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.567+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:31:30.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.575+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:31:30.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.575+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:30.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.575+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:30.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:30.575+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:32.601 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:31:32.627 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:31:35.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:35.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:35.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6753 cs=0 l=1 rev1=1 rx=0x7ff99409bb80 tx=0x7ff9940948a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:35.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 secure :-1 s=READY pgs=6753 cs=0 l=1 rev1=1 rx=0x7ff99409bb80 tx=0x7ff9940948a0).stop 2021-03-09T10:31:35.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980093f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:35.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980093f20 unknown :-1 s=CLOSED pgs=6753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:35.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:35.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:35.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:35.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026af0 con 0x7ff99c0adc20 2021-03-09T10:31:35.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c70 con 0x7ff99c0613a0 2021-03-09T10:31:35.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017270 con 0x7ff98000ac10 2021-03-09T10:31:35.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:35.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:35.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:35.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:35.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:35.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:35.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:35.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:35.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:35.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800854f0 con 0x7ff98000ac10 2021-03-09T10:31:35.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093400 con 0x7ff98000ac10 2021-03-09T10:31:35.581 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:35.571+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22749 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:35.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.571+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6761 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c0a2e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:35.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.575+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0872a0 con 0x7ff98000ac10 2021-03-09T10:31:35.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.575+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:31:35.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.579+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bcd0 con 0x7ff98000ac10 2021-03-09T10:31:35.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.579+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:35.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.579+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:35.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:35.579+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:40.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:40.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:40.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6761 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c0a2e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:40.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6761 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c0a2e20).stop 2021-03-09T10:31:40.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:40.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=6761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:40.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:40.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:40.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:40.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026af0 con 0x7ff99c0adc20 2021-03-09T10:31:40.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c70 con 0x7ff99c0613a0 2021-03-09T10:31:40.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017270 con 0x7ff980083100 2021-03-09T10:31:40.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:40.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:40.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:40.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:40.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.571+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:40.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.575+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:40.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.575+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:40.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.575+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:40.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.575+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:40.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.575+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093f20 con 0x7ff980083100 2021-03-09T10:31:40.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.575+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026c30 con 0x7ff980083100 2021-03-09T10:31:40.590 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:40.575+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22789 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:40.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.575+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6773 cs=0 l=1 rev1=1 rx=0x7ff994060800 tx=0x7ff9940681a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:40.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.579+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:31:40.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.579+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940411e0 con 0x7ff980083100 2021-03-09T10:31:40.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.583+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:31:40.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.583+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:40.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.583+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:40.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:40.583+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:45.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:45.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:45.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6773 cs=0 l=1 rev1=1 rx=0x7ff994060800 tx=0x7ff9940681a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:45.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6773 cs=0 l=1 rev1=1 rx=0x7ff994060800 tx=0x7ff9940681a0).stop 2021-03-09T10:31:45.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:45.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=CLOSED pgs=6773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:45.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:45.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:45.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:45.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022e60 con 0x7ff99c0adc20 2021-03-09T10:31:45.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c70 con 0x7ff99c0613a0 2021-03-09T10:31:45.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017270 con 0x7ff98000ac10 2021-03-09T10:31:45.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:45.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:45.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:45.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.575+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:45.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.579+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:45.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.579+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:45.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.579+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:45.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.579+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:45.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.579+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:45.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.579+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026af0 con 0x7ff98000ac10 2021-03-09T10:31:45.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.579+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093f20 con 0x7ff98000ac10 2021-03-09T10:31:45.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.579+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6787 cs=0 l=1 rev1=1 rx=0x7ff980038520 tx=0x7ff99408d1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:45.589 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:45.583+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22823 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:45.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.583+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff98000ac10 2021-03-09T10:31:45.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.587+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940b11c0 con 0x7ff98000ac10 2021-03-09T10:31:45.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.591+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff98000ac10 2021-03-09T10:31:45.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.591+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:45.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.591+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:45.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:45.591+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:50.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:50.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:50.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6787 cs=0 l=1 rev1=1 rx=0x7ff980038520 tx=0x7ff99408d1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:50.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6787 cs=0 l=1 rev1=1 rx=0x7ff980038520 tx=0x7ff99408d1f0).stop 2021-03-09T10:31:50.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:50.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=6787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:50.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:50.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:50.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:50.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c70 con 0x7ff99c0adc20 2021-03-09T10:31:50.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800179c0 con 0x7ff99c0613a0 2021-03-09T10:31:50.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093b90 con 0x7ff980083100 2021-03-09T10:31:50.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:50.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:50.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:50.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:50.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.583+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:50.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.587+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:50.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.587+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:50.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.587+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:50.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.587+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:50.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.587+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017270 con 0x7ff980083100 2021-03-09T10:31:50.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.587+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800854f0 con 0x7ff980083100 2021-03-09T10:31:50.595 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:50.587+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22863 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:50.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.587+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6794 cs=0 l=1 rev1=1 rx=0x7ff980094880 tx=0x7ff98c006650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:50.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.587+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980083100 2021-03-09T10:31:50.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.587+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2c00 con 0x7ff980083100 2021-03-09T10:31:50.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.591+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:31:50.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.591+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:50.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.595+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:50.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:50.595+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:31:55.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:55.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:55.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6794 cs=0 l=1 rev1=1 rx=0x7ff980094880 tx=0x7ff98c006650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:55.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6794 cs=0 l=1 rev1=1 rx=0x7ff980094880 tx=0x7ff98c006650).stop 2021-03-09T10:31:55.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:55.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=CLOSED pgs=6794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:55.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:55.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:55.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:55.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c70 con 0x7ff99c0adc20 2021-03-09T10:31:55.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093b90 con 0x7ff99c0613a0 2021-03-09T10:31:55.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800854f0 con 0x7ff98000ac10 2021-03-09T10:31:55.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:31:55.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:55.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:31:55.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:55.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:55.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:55.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:55.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:55.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:55.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.587+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980085630 con 0x7ff98000ac10 2021-03-09T10:31:55.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.591+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026af0 con 0x7ff98000ac10 2021-03-09T10:31:55.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.591+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6806 cs=0 l=1 rev1=1 rx=0x7ff98008e6f0 tx=0x7ff99402a180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:55.600 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:31:55.591+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22898 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:31:55.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.591+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940750b0 con 0x7ff98000ac10 2021-03-09T10:31:55.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.591+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400c380 con 0x7ff98000ac10 2021-03-09T10:31:55.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.599+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:31:55.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.599+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:55.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.599+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:55.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:31:55.599+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:00.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:00.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:00.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6806 cs=0 l=1 rev1=1 rx=0x7ff98008e6f0 tx=0x7ff99402a180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:00.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6806 cs=0 l=1 rev1=1 rx=0x7ff98008e6f0 tx=0x7ff99402a180).stop 2021-03-09T10:32:00.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:00.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=6806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:00.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:00.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:00.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:00.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c70 con 0x7ff99c0adc20 2021-03-09T10:32:00.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093b90 con 0x7ff99c0613a0 2021-03-09T10:32:00.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093d90 con 0x7ff980083100 2021-03-09T10:32:00.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:00.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:00.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:00.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:00.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:00.607 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:00.595+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22936 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:00.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:00.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:00.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:00.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:00.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980093ed0 con 0x7ff980083100 2021-03-09T10:32:00.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.595+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980085630 con 0x7ff980083100 2021-03-09T10:32:00.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.599+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6820 cs=0 l=1 rev1=1 rx=0x7ff99400c310 tx=0x7ff9940204b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:00.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.603+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980083100 2021-03-09T10:32:00.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.607+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980083100 2021-03-09T10:32:00.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.607+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:32:00.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.607+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:00.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.607+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:00.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:00.607+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:02.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:32:02.683 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:32:05.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:05.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:05.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6820 cs=0 l=1 rev1=1 rx=0x7ff99400c310 tx=0x7ff9940204b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:05.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6820 cs=0 l=1 rev1=1 rx=0x7ff99400c310 tx=0x7ff9940204b0).stop 2021-03-09T10:32:05.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:05.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=CLOSED pgs=6820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:05.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:05.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:05.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:05.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c70 con 0x7ff99c0adc20 2021-03-09T10:32:05.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980085510 con 0x7ff99c0613a0 2021-03-09T10:32:05.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980085650 con 0x7ff98000ac10 2021-03-09T10:32:05.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:05.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:05.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:05.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:05.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:05.607 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:05.599+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22971 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:05.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:05.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:05.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:05.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:05.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094060 con 0x7ff98000ac10 2021-03-09T10:32:05.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.599+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017270 con 0x7ff98000ac10 2021-03-09T10:32:05.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.603+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6830 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0797e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:05.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.603+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:32:05.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.611+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:32:05.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.611+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:32:05.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.611+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:05.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.611+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:05.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:05.611+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:10.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:10.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:10.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6830 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0797e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:10.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6830 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0797e0).stop 2021-03-09T10:32:10.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:10.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=6830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:10.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:10.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:10.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:10.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054c70 con 0x7ff99c0adc20 2021-03-09T10:32:10.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800536a0 con 0x7ff99c0613a0 2021-03-09T10:32:10.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980085510 con 0x7ff980083100 2021-03-09T10:32:10.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.599+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:10.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.603+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:10.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.603+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:10.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.603+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:10.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.603+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:10.611 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:10.603+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23010 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:10.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.607+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:10.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.607+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:10.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.607+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:10.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.607+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:10.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.607+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800941f0 con 0x7ff980083100 2021-03-09T10:32:10.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.607+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980093ed0 con 0x7ff980083100 2021-03-09T10:32:10.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.607+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6844 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff994036960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:10.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.611+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400c150 con 0x7ff980083100 2021-03-09T10:32:10.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.611+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400c150 con 0x7ff980083100 2021-03-09T10:32:10.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.615+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:32:10.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.615+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:10.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.615+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:10.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:10.615+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:15.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:15.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:15.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6844 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff994036960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:15.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6844 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff994036960).stop 2021-03-09T10:32:15.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:15.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=CLOSED pgs=6844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:15.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:15.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:15.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:15.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093ed0 con 0x7ff99c0adc20 2021-03-09T10:32:15.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980037f70 con 0x7ff99c0613a0 2021-03-09T10:32:15.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800380b0 con 0x7ff98000ac10 2021-03-09T10:32:15.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:15.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:15.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:15.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:15.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:15.618 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:15.606+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23045 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:15.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:15.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:15.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:15.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:15.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980053670 con 0x7ff98000ac10 2021-03-09T10:32:15.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.606+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800941f0 con 0x7ff98000ac10 2021-03-09T10:32:15.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.610+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6850 cs=0 l=1 rev1=1 rx=0x7ff9940135d0 tx=0x7ff994081670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:15.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.610+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff98000ac10 2021-03-09T10:32:15.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.614+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff98000ac10 2021-03-09T10:32:15.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.614+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff98000ac10 2021-03-09T10:32:15.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.614+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:15.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.614+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:15.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:15.614+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:20.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:20.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:20.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6850 cs=0 l=1 rev1=1 rx=0x7ff9940135d0 tx=0x7ff994081670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:20.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6850 cs=0 l=1 rev1=1 rx=0x7ff9940135d0 tx=0x7ff994081670).stop 2021-03-09T10:32:20.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:20.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=6850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:20.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:20.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:20.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:20.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053390 con 0x7ff99c0adc20 2021-03-09T10:32:20.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053800 con 0x7ff99c0613a0 2021-03-09T10:32:20.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:20.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093ed0 con 0x7ff980083100 2021-03-09T10:32:20.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:20.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:20.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:20.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:20.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:20.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:20.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:20.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:20.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800941f0 con 0x7ff980083100 2021-03-09T10:32:20.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094060 con 0x7ff980083100 2021-03-09T10:32:20.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.610+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6857 cs=0 l=1 rev1=1 rx=0x7ff98c0052d0 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:20.623 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:20.614+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23084 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:20.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.614+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff980083100 2021-03-09T10:32:20.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.614+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2c00 con 0x7ff980083100 2021-03-09T10:32:20.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.618+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c07dd90 con 0x7ff980083100 2021-03-09T10:32:20.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.618+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:20.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.618+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:20.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:20.618+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:25.623 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:25.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:25.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6857 cs=0 l=1 rev1=1 rx=0x7ff98c0052d0 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:25.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6857 cs=0 l=1 rev1=1 rx=0x7ff98c0052d0 tx=0x7ff98c003e80).stop 2021-03-09T10:32:25.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:25.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=CLOSED pgs=6857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:25.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:25.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:25.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:25.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053390 con 0x7ff99c0adc20 2021-03-09T10:32:25.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053800 con 0x7ff99c0613a0 2021-03-09T10:32:25.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094070 con 0x7ff98000ac10 2021-03-09T10:32:25.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:25.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:25.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:25.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:25.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:25.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:25.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:25.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:25.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:25.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094380 con 0x7ff98000ac10 2021-03-09T10:32:25.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980053670 con 0x7ff98000ac10 2021-03-09T10:32:25.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.614+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6869 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff9940845f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:25.641 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:25.618+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23119 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:25.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.618+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095120 con 0x7ff98000ac10 2021-03-09T10:32:25.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.618+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff98000ac10 2021-03-09T10:32:25.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.622+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff98000ac10 2021-03-09T10:32:25.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.622+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:25.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.622+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:25.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:25.622+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:30.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:30.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:30.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6869 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff9940845f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:30.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6869 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff9940845f0).stop 2021-03-09T10:32:30.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:30.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=6869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:30.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:30.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:30.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:30.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092fe0 con 0x7ff99c0adc20 2021-03-09T10:32:30.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005200 con 0x7ff99c0613a0 2021-03-09T10:32:30.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093ed0 con 0x7ff980083100 2021-03-09T10:32:30.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:30.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:30.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:30.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:30.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:30.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:30.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:30.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:30.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:30.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094010 con 0x7ff980083100 2021-03-09T10:32:30.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.618+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094380 con 0x7ff980083100 2021-03-09T10:32:30.634 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:30.622+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23158 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:30.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.622+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6879 cs=0 l=1 rev1=1 rx=0x7ff994011480 tx=0x7ff994081670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:30.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.622+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980083100 2021-03-09T10:32:30.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.622+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400d1c0 con 0x7ff980083100 2021-03-09T10:32:30.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.630+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700e0 con 0x7ff980083100 2021-03-09T10:32:30.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.630+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:30.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.630+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:30.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:30.630+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:32.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:32:32.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:32:35.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:35.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:35.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6879 cs=0 l=1 rev1=1 rx=0x7ff994011480 tx=0x7ff994081670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:35.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 secure :-1 s=READY pgs=6879 cs=0 l=1 rev1=1 rx=0x7ff994011480 tx=0x7ff994081670).stop 2021-03-09T10:32:35.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000c470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:35.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000c470 unknown :-1 s=CLOSED pgs=6879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:35.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:35.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:35.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:35.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093e90 con 0x7ff99c0adc20 2021-03-09T10:32:35.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005200 con 0x7ff99c0613a0 2021-03-09T10:32:35.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce40 con 0x7ff98000ac10 2021-03-09T10:32:35.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:35.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:35.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:35.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:35.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:35.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:35.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:35.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:35.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:35.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008cf80 con 0x7ff98000ac10 2021-03-09T10:32:35.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.622+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094010 con 0x7ff98000ac10 2021-03-09T10:32:35.634 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:35.626+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23194 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:35.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.626+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6891 cs=0 l=1 rev1=1 rx=0x7ff994021040 tx=0x7ff99400ca10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:35.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.626+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994085e10 con 0x7ff98000ac10 2021-03-09T10:32:35.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.630+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994085e10 con 0x7ff98000ac10 2021-03-09T10:32:35.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.634+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:32:35.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.634+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:35.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.634+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:35.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:35.634+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:40.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:40.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:40.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6891 cs=0 l=1 rev1=1 rx=0x7ff994021040 tx=0x7ff99400ca10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:40.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6891 cs=0 l=1 rev1=1 rx=0x7ff994021040 tx=0x7ff99400ca10).stop 2021-03-09T10:32:40.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:40.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=6891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:40.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:40.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:40.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:40.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ce20 con 0x7ff99c0adc20 2021-03-09T10:32:40.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800051b0 con 0x7ff99c0613a0 2021-03-09T10:32:40.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094380 con 0x7ff980083100 2021-03-09T10:32:40.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:40.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:40.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:40.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:40.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:40.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:40.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:40.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:40.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:40.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800944c0 con 0x7ff980083100 2021-03-09T10:32:40.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008cf80 con 0x7ff980083100 2021-03-09T10:32:40.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023380 secure :-1 s=READY pgs=6901 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994029120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:40.642 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:40.630+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23232 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:40.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994064e80 con 0x7ff980083100 2021-03-09T10:32:40.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.630+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994085e10 con 0x7ff980083100 2021-03-09T10:32:40.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.638+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:32:40.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.638+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:40.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.638+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:40.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:40.638+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:45.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:45.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:45.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023380 secure :-1 s=READY pgs=6901 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994029120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:45.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023380 secure :-1 s=READY pgs=6901 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994029120).stop 2021-03-09T10:32:45.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:45.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023380 unknown :-1 s=CLOSED pgs=6901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:45.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:45.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:45.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:45.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052660 con 0x7ff99c0adc20 2021-03-09T10:32:45.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046420 con 0x7ff99c0613a0 2021-03-09T10:32:45.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c470 con 0x7ff98000ac10 2021-03-09T10:32:45.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:45.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:45.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:45.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:45.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:45.642 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:45.634+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23267 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:45.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:45.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:45.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:45.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:45.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005010 con 0x7ff98000ac10 2021-03-09T10:32:45.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800941f0 con 0x7ff98000ac10 2021-03-09T10:32:45.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.634+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6913 cs=0 l=1 rev1=1 rx=0x7ff98c00c010 tx=0x7ff98c005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:45.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.638+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04bd70 con 0x7ff98000ac10 2021-03-09T10:32:45.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.638+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:32:45.650 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.642+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bd70 con 0x7ff98000ac10 2021-03-09T10:32:45.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.642+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:45.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.646+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:45.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:45.646+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:50.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:50.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:50.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6913 cs=0 l=1 rev1=1 rx=0x7ff98c00c010 tx=0x7ff98c005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:50.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 secure :-1 s=READY pgs=6913 cs=0 l=1 rev1=1 rx=0x7ff98c00c010 tx=0x7ff98c005110).stop 2021-03-09T10:32:50.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98000bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:50.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98000bf80 unknown :-1 s=CLOSED pgs=6913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:50.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:50.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:50.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:50.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c470 con 0x7ff99c0adc20 2021-03-09T10:32:50.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016f10 con 0x7ff99c0613a0 2021-03-09T10:32:50.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023040 con 0x7ff980083100 2021-03-09T10:32:50.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:50.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:50.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:50.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:50.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:50.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:50.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:50.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:50.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:50.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980023180 con 0x7ff980083100 2021-03-09T10:32:50.648 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:50.638+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23306 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:50.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.638+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005010 con 0x7ff980083100 2021-03-09T10:32:50.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.642+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cc60 secure :-1 s=READY pgs=6923 cs=0 l=1 rev1=1 rx=0x7ff98c0052a0 tx=0x7ff98c0a91d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:50.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.646+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079c90 con 0x7ff980083100 2021-03-09T10:32:50.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.646+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c072970 con 0x7ff980083100 2021-03-09T10:32:50.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.650+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072ae0 con 0x7ff980083100 2021-03-09T10:32:50.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.650+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:50.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.650+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:50.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:50.650+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:32:55.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:55.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:55.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cc60 secure :-1 s=READY pgs=6923 cs=0 l=1 rev1=1 rx=0x7ff98c0052a0 tx=0x7ff98c0a91d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:55.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cc60 secure :-1 s=READY pgs=6923 cs=0 l=1 rev1=1 rx=0x7ff98c0052a0 tx=0x7ff98c0a91d0).stop 2021-03-09T10:32:55.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008cc60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:55.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cc60 unknown :-1 s=CLOSED pgs=6923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:55.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:55.650 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:55.650 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:55.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0adc20 2021-03-09T10:32:55.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800051b0 con 0x7ff99c0613a0 2021-03-09T10:32:55.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000bf80 con 0x7ff98000ac10 2021-03-09T10:32:55.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:32:55.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:55.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:32:55.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:55.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:55.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:55.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:55.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:55.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:55.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000c0c0 con 0x7ff98000ac10 2021-03-09T10:32:55.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800944c0 con 0x7ff98000ac10 2021-03-09T10:32:55.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 secure :-1 s=READY pgs=6933 cs=0 l=1 rev1=1 rx=0x7ff99406e040 tx=0x7ff99408dff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:55.659 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:32:55.642+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23341 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:32:55.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.642+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994084560 con 0x7ff98000ac10 2021-03-09T10:32:55.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.646+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994084560 con 0x7ff98000ac10 2021-03-09T10:32:55.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.650+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:32:55.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.650+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:55.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.650+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:55.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:32:55.650+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:00.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:00.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:00.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 secure :-1 s=READY pgs=6933 cs=0 l=1 rev1=1 rx=0x7ff99406e040 tx=0x7ff99408dff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:00.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 secure :-1 s=READY pgs=6933 cs=0 l=1 rev1=1 rx=0x7ff99406e040 tx=0x7ff99408dff0).stop 2021-03-09T10:33:00.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:00.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 unknown :-1 s=CLOSED pgs=6933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:00.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:00.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:00.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:00.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000bf80 con 0x7ff99c0adc20 2021-03-09T10:33:00.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0613a0 2021-03-09T10:33:00.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800051b0 con 0x7ff980083100 2021-03-09T10:33:00.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:00.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:00.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:00.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:00.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:00.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:00.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:00.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:00.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:00.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094660 con 0x7ff980083100 2021-03-09T10:33:00.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000c0c0 con 0x7ff980083100 2021-03-09T10:33:00.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.646+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ca70 secure :-1 s=READY pgs=6943 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99407a230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:00.658 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:00.646+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23380 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:00.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.650+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940700e0 con 0x7ff980083100 2021-03-09T10:33:00.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.650+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994016340 con 0x7ff980083100 2021-03-09T10:33:00.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.654+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700e0 con 0x7ff980083100 2021-03-09T10:33:00.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.654+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:00.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.654+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:00.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:00.654+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:02.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:33:02.802 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:33:05.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:33:05.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:05.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ca70 secure :-1 s=READY pgs=6943 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99407a230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:05.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ca70 secure :-1 s=READY pgs=6943 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99407a230).stop 2021-03-09T10:33:05.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008ca70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:05.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008ca70 unknown :-1 s=CLOSED pgs=6943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:05.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:05.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:05.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:05.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001e8e0 con 0x7ff99c0adc20 2021-03-09T10:33:05.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0613a0 2021-03-09T10:33:05.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800051b0 con 0x7ff98000ac10 2021-03-09T10:33:05.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:05.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:05.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:05.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:05.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:05.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:05.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:05.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:05.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:05.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000bf80 con 0x7ff98000ac10 2021-03-09T10:33:05.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094660 con 0x7ff98000ac10 2021-03-09T10:33:05.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.650+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 secure :-1 s=READY pgs=6953 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff99402b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:05.663 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:05.654+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23415 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:05.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.654+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994014120 con 0x7ff98000ac10 2021-03-09T10:33:05.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.654+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:33:05.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.658+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700e0 con 0x7ff98000ac10 2021-03-09T10:33:05.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.658+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:05.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.662+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:05.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:05.662+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:10.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:10.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:10.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 secure :-1 s=READY pgs=6953 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff99402b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:10.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 secure :-1 s=READY pgs=6953 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff99402b000).stop 2021-03-09T10:33:10.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:10.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023310 unknown :-1 s=CLOSED pgs=6953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:10.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:10.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:10.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:10.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c2b0 con 0x7ff99c0adc20 2021-03-09T10:33:10.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980089210 con 0x7ff99c0613a0 2021-03-09T10:33:10.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019e00 con 0x7ff980083100 2021-03-09T10:33:10.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:10.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:10.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:10.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:10.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:10.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:10.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:10.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:10.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.654+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:10.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.658+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:33:10.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.658+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980023180 con 0x7ff980083100 2021-03-09T10:33:10.667 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:10.658+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23454 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:10.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.658+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008c9d0 secure :-1 s=READY pgs=6966 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c04bed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:10.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.658+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:33:10.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.658+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980083100 2021-03-09T10:33:10.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.662+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:33:10.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.662+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:10.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.662+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:10.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:10.662+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:15.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:15.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:15.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008c9d0 secure :-1 s=READY pgs=6966 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c04bed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:15.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008c9d0 secure :-1 s=READY pgs=6966 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c04bed0).stop 2021-03-09T10:33:15.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008c9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:15.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008c9d0 unknown :-1 s=CLOSED pgs=6966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:15.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:15.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:15.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:15.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800383e0 con 0x7ff99c0adc20 2021-03-09T10:33:15.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005880 con 0x7ff99c0613a0 2021-03-09T10:33:15.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800051b0 con 0x7ff98000ac10 2021-03-09T10:33:15.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:15.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:15.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:15.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:15.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:15.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:15.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:15.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:15.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:15.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980094660 con 0x7ff98000ac10 2021-03-09T10:33:15.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000bf80 con 0x7ff98000ac10 2021-03-09T10:33:15.670 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:15.662+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23489 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:15.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.662+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 secure :-1 s=READY pgs=6977 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff994001f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:15.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400ef00 con 0x7ff98000ac10 2021-03-09T10:33:15.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:33:15.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.670+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994084180 con 0x7ff98000ac10 2021-03-09T10:33:15.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.670+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:15.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.670+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:15.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:15.670+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:20.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:20.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:20.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 secure :-1 s=READY pgs=6977 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff994001f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:20.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 secure :-1 s=READY pgs=6977 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff994001f20).stop 2021-03-09T10:33:20.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:20.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 unknown :-1 s=CLOSED pgs=6977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:20.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:20.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:20.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:20.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000cb50 con 0x7ff99c0adc20 2021-03-09T10:33:20.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005880 con 0x7ff99c0613a0 2021-03-09T10:33:20.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c120 con 0x7ff980083100 2021-03-09T10:33:20.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:20.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:20.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:20.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:20.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:20.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:20.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:20.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:20.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:20.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800051b0 con 0x7ff980083100 2021-03-09T10:33:20.675 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:20.666+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23528 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:20.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980094660 con 0x7ff980083100 2021-03-09T10:33:20.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019c30 secure :-1 s=READY pgs=6987 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff994034070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:20.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:33:20.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.670+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:33:20.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.674+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700e0 con 0x7ff980083100 2021-03-09T10:33:20.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.674+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:20.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.674+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:20.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:20.674+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:25.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980019c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:25.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980019c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:25.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019c30 secure :-1 s=READY pgs=6987 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff994034070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:25.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019c30 secure :-1 s=READY pgs=6987 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff994034070).stop 2021-03-09T10:33:25.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980019c30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:25.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980019c30 unknown :-1 s=CLOSED pgs=6987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:25.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:25.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:25.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:25.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800920c0 con 0x7ff99c0adc20 2021-03-09T10:33:25.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005880 con 0x7ff99c0613a0 2021-03-09T10:33:25.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023670 con 0x7ff98000ac10 2021-03-09T10:33:25.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:25.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:25.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:25.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:25.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.666+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:25.678 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:25.670+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23563 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:25.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.670+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:25.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.670+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:25.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.670+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:25.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.670+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:25.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.670+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008c930 con 0x7ff98000ac10 2021-03-09T10:33:25.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.670+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800051b0 con 0x7ff98000ac10 2021-03-09T10:33:25.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.670+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 secure :-1 s=READY pgs=7003 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994009d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:25.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.674+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:33:25.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.674+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:33:25.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.678+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99406d590 con 0x7ff98000ac10 2021-03-09T10:33:25.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.678+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:25.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.678+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:25.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:25.678+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:30.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:30.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:30.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 secure :-1 s=READY pgs=7003 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994009d80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:30.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 secure :-1 s=READY pgs=7003 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994009d80).stop 2021-03-09T10:33:30.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:30.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023180 unknown :-1 s=CLOSED pgs=7003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:30.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:30.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:30.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:30.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005880 con 0x7ff99c0adc20 2021-03-09T10:33:30.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092090 con 0x7ff99c0613a0 2021-03-09T10:33:30.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cfb0 con 0x7ff980083100 2021-03-09T10:33:30.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:30.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:30.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:30.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:30.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:30.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:30.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:30.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:30.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:30.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019a40 con 0x7ff980083100 2021-03-09T10:33:30.683 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:30.674+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23602 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:30.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008c930 con 0x7ff980083100 2021-03-09T10:33:30.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.674+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cac0 secure :-1 s=READY pgs=7011 cs=0 l=1 rev1=1 rx=0x7ff994085390 tx=0x7ff994029190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:30.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.678+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff980083100 2021-03-09T10:33:30.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.678+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081340 con 0x7ff980083100 2021-03-09T10:33:30.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.682+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994084180 con 0x7ff980083100 2021-03-09T10:33:30.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.682+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:30.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.682+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:30.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:30.682+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:32.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:33:32.852 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:33:35.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:35.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:35.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cac0 secure :-1 s=READY pgs=7011 cs=0 l=1 rev1=1 rx=0x7ff994085390 tx=0x7ff994029190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:35.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cac0 secure :-1 s=READY pgs=7011 cs=0 l=1 rev1=1 rx=0x7ff994085390 tx=0x7ff994029190).stop 2021-03-09T10:33:35.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008cac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:35.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008cac0 unknown :-1 s=CLOSED pgs=7011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:35.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:35.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:35.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:35.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cfb0 con 0x7ff99c0adc20 2021-03-09T10:33:35.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092090 con 0x7ff99c0613a0 2021-03-09T10:33:35.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005880 con 0x7ff98000ac10 2021-03-09T10:33:35.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:35.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:35.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:35.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:35.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:35.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:35.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:35.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:35.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:35.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980023180 con 0x7ff98000ac10 2021-03-09T10:33:35.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019a40 con 0x7ff98000ac10 2021-03-09T10:33:35.688 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:35.677+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23637 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:35.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019bd0 secure :-1 s=READY pgs=7017 cs=0 l=1 rev1=1 rx=0x7ff994060740 tx=0x7ff9940018d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:35.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081340 con 0x7ff98000ac10 2021-03-09T10:33:35.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff98000ac10 2021-03-09T10:33:35.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.685+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff98000ac10 2021-03-09T10:33:35.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.685+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:35.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.685+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:35.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:35.685+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:40.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:40.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:40.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019bd0 secure :-1 s=READY pgs=7017 cs=0 l=1 rev1=1 rx=0x7ff994060740 tx=0x7ff9940018d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:40.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019bd0 secure :-1 s=READY pgs=7017 cs=0 l=1 rev1=1 rx=0x7ff994060740 tx=0x7ff9940018d0).stop 2021-03-09T10:33:40.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:40.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019bd0 unknown :-1 s=CLOSED pgs=7017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:40.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:40.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:40.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:40.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092090 con 0x7ff99c0adc20 2021-03-09T10:33:40.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:40.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005880 con 0x7ff99c0613a0 2021-03-09T10:33:40.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c8e0 con 0x7ff980083100 2021-03-09T10:33:40.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:40.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:40.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:40.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:40.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:40.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:40.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:40.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:40.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008ca20 con 0x7ff980083100 2021-03-09T10:33:40.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980023180 con 0x7ff980083100 2021-03-09T10:33:40.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 secure :-1 s=READY pgs=7029 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994068a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:40.693 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:40.681+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23676 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:40.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994060230 con 0x7ff980083100 2021-03-09T10:33:40.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.681+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994060390 con 0x7ff980083100 2021-03-09T10:33:40.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.689+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:33:40.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.689+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:40.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.689+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:40.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:40.689+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:45.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:45.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:45.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 secure :-1 s=READY pgs=7029 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994068a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:45.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 secure :-1 s=READY pgs=7029 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994068a70).stop 2021-03-09T10:33:45.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:45.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 unknown :-1 s=CLOSED pgs=7029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:45.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:45.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:45.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:45.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008c8e0 con 0x7ff99c0adc20 2021-03-09T10:33:45.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005880 con 0x7ff99c0613a0 2021-03-09T10:33:45.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092090 con 0x7ff98000ac10 2021-03-09T10:33:45.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:45.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:45.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:45.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:45.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:45.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:45.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:45.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:45.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:45.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980023180 con 0x7ff98000ac10 2021-03-09T10:33:45.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.685+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ca20 con 0x7ff98000ac10 2021-03-09T10:33:45.698 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:45.689+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23710 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:45.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.689+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199f0 secure :-1 s=READY pgs=7042 cs=0 l=1 rev1=1 rx=0x7ff99407a320 tx=0x7ff994094f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:45.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.689+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400c330 con 0x7ff98000ac10 2021-03-09T10:33:45.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.693+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400c150 con 0x7ff98000ac10 2021-03-09T10:33:45.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.693+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700b0 con 0x7ff98000ac10 2021-03-09T10:33:45.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.693+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:45.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.697+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:45.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:45.697+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:50.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800199f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:50.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800199f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:50.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199f0 secure :-1 s=READY pgs=7042 cs=0 l=1 rev1=1 rx=0x7ff99407a320 tx=0x7ff994094f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:50.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199f0 secure :-1 s=READY pgs=7042 cs=0 l=1 rev1=1 rx=0x7ff99407a320 tx=0x7ff994094f00).stop 2021-03-09T10:33:50.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800199f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:50.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199f0 unknown :-1 s=CLOSED pgs=7042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:50.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:50.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:50.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:50.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800893b0 con 0x7ff99c0adc20 2021-03-09T10:33:50.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b470 con 0x7ff99c0613a0 2021-03-09T10:33:50.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980088a60 con 0x7ff980083100 2021-03-09T10:33:50.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:50.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:50.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:50.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:50.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:50.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:50.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:50.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:50.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:50.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008c8e0 con 0x7ff980083100 2021-03-09T10:33:50.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980023180 con 0x7ff980083100 2021-03-09T10:33:50.702 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:50.693+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23750 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:50.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.693+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 secure :-1 s=READY pgs=7051 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406d500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:50.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.697+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980083100 2021-03-09T10:33:50.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.697+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994085a70 con 0x7ff980083100 2021-03-09T10:33:50.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.701+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060230 con 0x7ff980083100 2021-03-09T10:33:50.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.701+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:50.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.701+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:50.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:50.701+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:33:55.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:55.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:55.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 secure :-1 s=READY pgs=7051 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406d500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:55.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 secure :-1 s=READY pgs=7051 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99406d500).stop 2021-03-09T10:33:55.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:55.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 unknown :-1 s=CLOSED pgs=7051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:55.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:55.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:55.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:55.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a760 con 0x7ff99c0adc20 2021-03-09T10:33:55.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039470 con 0x7ff99c0613a0 2021-03-09T10:33:55.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022860 con 0x7ff98000ac10 2021-03-09T10:33:55.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:33:55.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:55.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:33:55.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:55.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:55.705 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:33:55.697+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23785 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:33:55.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:55.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:55.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:55.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:55.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019cf0 con 0x7ff98000ac10 2021-03-09T10:33:55.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008c8e0 con 0x7ff98000ac10 2021-03-09T10:33:55.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.697+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019800 secure :-1 s=READY pgs=7067 cs=0 l=1 rev1=1 rx=0x7ff994038970 tx=0x7ff994086e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:55.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.701+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994085660 con 0x7ff98000ac10 2021-03-09T10:33:55.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.705+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994085660 con 0x7ff98000ac10 2021-03-09T10:33:55.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.705+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99404c120 con 0x7ff98000ac10 2021-03-09T10:33:55.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.705+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:55.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.705+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:55.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:33:55.705+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:00.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:00.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:00.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019800 secure :-1 s=READY pgs=7067 cs=0 l=1 rev1=1 rx=0x7ff994038970 tx=0x7ff994086e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:00.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019800 secure :-1 s=READY pgs=7067 cs=0 l=1 rev1=1 rx=0x7ff994038970 tx=0x7ff994086e50).stop 2021-03-09T10:34:00.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:00.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019800 unknown :-1 s=CLOSED pgs=7067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:00.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:00.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:00.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:00.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039550 con 0x7ff99c0adc20 2021-03-09T10:34:00.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029fe0 con 0x7ff99c0613a0 2021-03-09T10:34:00.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022810 con 0x7ff980083100 2021-03-09T10:34:00.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:00.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:00.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:00.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:00.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:00.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:00.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:00.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:00.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:00.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022950 con 0x7ff980083100 2021-03-09T10:34:00.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.701+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:34:00.711 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:00.705+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23824 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:00.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.705+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 secure :-1 s=READY pgs=7071 cs=0 l=1 rev1=1 rx=0x7ff98c004230 tx=0x7ff98c04bf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:00.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.709+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:34:00.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.709+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980083100 2021-03-09T10:34:00.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.709+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980083100 2021-03-09T10:34:00.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.709+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:00.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.713+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:00.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:00.713+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:02.892 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:34:02.895 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:34:05.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:05.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:05.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 secure :-1 s=READY pgs=7071 cs=0 l=1 rev1=1 rx=0x7ff98c004230 tx=0x7ff98c04bf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:05.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 secure :-1 s=READY pgs=7071 cs=0 l=1 rev1=1 rx=0x7ff98c004230 tx=0x7ff98c04bf40).stop 2021-03-09T10:34:05.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980023310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:05.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980023310 unknown :-1 s=CLOSED pgs=7071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:05.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:05.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:05.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:05.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039550 con 0x7ff99c0adc20 2021-03-09T10:34:05.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0613a0 2021-03-09T10:34:05.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800227c0 con 0x7ff98000ac10 2021-03-09T10:34:05.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:05.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:05.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:05.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:05.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.705+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:05.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.709+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:05.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.709+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:05.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.709+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:05.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.709+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:05.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.709+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800194c0 con 0x7ff98000ac10 2021-03-09T10:34:05.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.709+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022950 con 0x7ff98000ac10 2021-03-09T10:34:05.718 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:05.709+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23858 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:05.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.709+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7083 cs=0 l=1 rev1=1 rx=0x7ff98c005f90 tx=0x7ff98c07dda0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:05.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.713+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:34:05.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.713+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:34:05.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.717+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:34:05.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.717+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:05.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.717+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:05.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:05.717+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:10.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:10.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:10.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7083 cs=0 l=1 rev1=1 rx=0x7ff98c005f90 tx=0x7ff98c07dda0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:10.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7083 cs=0 l=1 rev1=1 rx=0x7ff98c005f90 tx=0x7ff98c07dda0).stop 2021-03-09T10:34:10.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:10.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=CLOSED pgs=7083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:10.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:10.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:10.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:10.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177b0 con 0x7ff99c0adc20 2021-03-09T10:34:10.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039550 con 0x7ff99c0613a0 2021-03-09T10:34:10.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022b00 con 0x7ff980083100 2021-03-09T10:34:10.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:10.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:10.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:10.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:10.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.709+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:10.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.713+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:10.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.713+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:10.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.713+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:10.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.713+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:10.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.713+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:34:10.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.713+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800194c0 con 0x7ff980083100 2021-03-09T10:34:10.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.713+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7094 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c0a2f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:10.720 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:10.713+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23898 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:10.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.717+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980083100 2021-03-09T10:34:10.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.721+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980083100 2021-03-09T10:34:10.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.721+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980083100 2021-03-09T10:34:10.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.721+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:10.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.721+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:10.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:10.721+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:15.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.713+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:15.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.713+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:15.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.713+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7094 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c0a2f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:15.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.713+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7094 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c0a2f40).stop 2021-03-09T10:34:15.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.713+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:15.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.713+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:15.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:15.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800227c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:15.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:15.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000eb20 con 0x7ff99c0adc20 2021-03-09T10:34:15.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177b0 con 0x7ff99c0613a0 2021-03-09T10:34:15.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022cb0 con 0x7ff98000ac10 2021-03-09T10:34:15.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:15.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:15.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:15.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:15.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.717+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800227c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:15.724 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:15.717+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23932 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:15.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.721+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:15.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.721+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:15.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.721+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:15.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.721+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:15.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.721+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019410 con 0x7ff98000ac10 2021-03-09T10:34:15.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.721+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022ff0 con 0x7ff98000ac10 2021-03-09T10:34:15.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.721+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800227c0 secure :-1 s=READY pgs=7108 cs=0 l=1 rev1=1 rx=0x7ff98c07e490 tx=0x7ff98c07df70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:15.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.725+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c017ec0 con 0x7ff98000ac10 2021-03-09T10:34:15.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.725+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c017ec0 con 0x7ff98000ac10 2021-03-09T10:34:15.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.725+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:34:15.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.725+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:15.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.729+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:15.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:15.729+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:20.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.717+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800227c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:20.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.717+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800227c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:20.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.717+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800227c0 secure :-1 s=READY pgs=7108 cs=0 l=1 rev1=1 rx=0x7ff98c07e490 tx=0x7ff98c07df70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:20.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800227c0 secure :-1 s=READY pgs=7108 cs=0 l=1 rev1=1 rx=0x7ff98c07e490 tx=0x7ff98c07df70).stop 2021-03-09T10:34:20.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800227c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:20.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800227c0 unknown :-1 s=CLOSED pgs=7108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:20.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:20.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:20.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:20.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011400 con 0x7ff99c0adc20 2021-03-09T10:34:20.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022cb0 con 0x7ff99c0613a0 2021-03-09T10:34:20.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800177b0 con 0x7ff980083100 2021-03-09T10:34:20.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:20.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:20.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:20.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:20.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:20.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:20.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:20.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:20.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:20.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:34:20.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019cf0 con 0x7ff980083100 2021-03-09T10:34:20.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7113 cs=0 l=1 rev1=1 rx=0x7ff99403f040 tx=0x7ff994081d70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:20.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.721+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994085840 con 0x7ff980083100 2021-03-09T10:34:20.730 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:20.721+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23972 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:20.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.725+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980083100 2021-03-09T10:34:20.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.729+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:34:20.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.729+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:20.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.729+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:20.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:20.729+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:25.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.721+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:25.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:25.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7113 cs=0 l=1 rev1=1 rx=0x7ff99403f040 tx=0x7ff994081d70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:25.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7113 cs=0 l=1 rev1=1 rx=0x7ff99403f040 tx=0x7ff994081d70).stop 2021-03-09T10:34:25.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:25.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:25.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800195a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:25.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:25.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:25.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003ce0 con 0x7ff99c0adc20 2021-03-09T10:34:25.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053480 con 0x7ff99c0613a0 2021-03-09T10:34:25.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017950 con 0x7ff98000ac10 2021-03-09T10:34:25.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:25.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:25.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:25.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800195a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:25.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:25.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:25.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:25.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:25.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:25.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019a90 con 0x7ff98000ac10 2021-03-09T10:34:25.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.725+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022ff0 con 0x7ff98000ac10 2021-03-09T10:34:25.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.729+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800195a0 secure :-1 s=READY pgs=7127 cs=0 l=1 rev1=1 rx=0x7ff994068e90 tx=0x7ff9940858b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:25.734 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:25.729+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24006 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:25.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.733+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff98000ac10 2021-03-09T10:34:25.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.733+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940701d0 con 0x7ff98000ac10 2021-03-09T10:34:25.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.733+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff98000ac10 2021-03-09T10:34:25.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.733+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:25.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.737+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:25.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:25.737+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:30.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800195a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:30.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800195a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:30.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800195a0 secure :-1 s=READY pgs=7127 cs=0 l=1 rev1=1 rx=0x7ff994068e90 tx=0x7ff9940858b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:30.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800195a0 secure :-1 s=READY pgs=7127 cs=0 l=1 rev1=1 rx=0x7ff994068e90 tx=0x7ff9940858b0).stop 2021-03-09T10:34:30.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800195a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:30.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800195a0 unknown :-1 s=CLOSED pgs=7127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:30.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:30.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:30.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:30.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003cb0 con 0x7ff99c0adc20 2021-03-09T10:34:30.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014800 con 0x7ff99c0613a0 2021-03-09T10:34:30.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980053480 con 0x7ff980083100 2021-03-09T10:34:30.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:30.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:30.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:30.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:30.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:30.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:30.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:30.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:30.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.729+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:30.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.733+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:34:30.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.733+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019a90 con 0x7ff980083100 2021-03-09T10:34:30.739 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:30.733+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24046 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:30.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.733+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7136 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff994095190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:30.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.741+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b02c0 con 0x7ff980083100 2021-03-09T10:34:30.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.741+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940b0420 con 0x7ff980083100 2021-03-09T10:34:30.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.741+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99407bbb0 con 0x7ff980083100 2021-03-09T10:34:30.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.741+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:30.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.741+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:30.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:30.741+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:33.107 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:34:33.110 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:34:35.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:34:35.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:35.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7136 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff994095190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:35.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7136 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff994095190).stop 2021-03-09T10:34:35.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:35.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:35.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:35.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:35.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:35.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017b00 con 0x7ff99c0adc20 2021-03-09T10:34:35.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017c40 con 0x7ff99c0613a0 2021-03-09T10:34:35.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800195a0 con 0x7ff98000ac10 2021-03-09T10:34:35.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:35.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:35.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:35.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:35.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.733+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:35.741 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:35.737+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24080 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:35.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.737+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:35.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.737+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:35.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.737+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:35.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.737+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:35.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.737+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800196e0 con 0x7ff98000ac10 2021-03-09T10:34:35.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.737+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019410 con 0x7ff98000ac10 2021-03-09T10:34:35.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.737+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017610 secure :-1 s=READY pgs=7149 cs=0 l=1 rev1=1 rx=0x7ff98c0a1010 tx=0x7ff98c08f3b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:35.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.741+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08f130 con 0x7ff98000ac10 2021-03-09T10:34:35.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.741+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08f130 con 0x7ff98000ac10 2021-03-09T10:34:35.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.745+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:34:35.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.745+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:35.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.745+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:35.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:35.745+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:40.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:40.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:40.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017610 secure :-1 s=READY pgs=7149 cs=0 l=1 rev1=1 rx=0x7ff98c0a1010 tx=0x7ff98c08f3b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:40.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017610 secure :-1 s=READY pgs=7149 cs=0 l=1 rev1=1 rx=0x7ff98c0a1010 tx=0x7ff98c08f3b0).stop 2021-03-09T10:34:40.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:40.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017610 unknown :-1 s=CLOSED pgs=7149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:40.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:40.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:40.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:40.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017ca0 con 0x7ff99c0adc20 2021-03-09T10:34:40.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022700 con 0x7ff99c0613a0 2021-03-09T10:34:40.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022840 con 0x7ff980083100 2021-03-09T10:34:40.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:40.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:40.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:40.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:40.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.737+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:40.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.741+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:40.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.741+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:40.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.741+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:40.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.741+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:40.747 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:40.741+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24120 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:40.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.741+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019870 con 0x7ff980083100 2021-03-09T10:34:40.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.741+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:34:40.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.741+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7158 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940953c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:40.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.745+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:34:40.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.745+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940701d0 con 0x7ff980083100 2021-03-09T10:34:40.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.749+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:34:40.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.749+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:40.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.749+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:40.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:40.749+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:45.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:45.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:45.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7158 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940953c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:45.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7158 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940953c0).stop 2021-03-09T10:34:45.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:45.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:45.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:45.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:45.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:45.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0adc20 2021-03-09T10:34:45.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017610 con 0x7ff99c0613a0 2021-03-09T10:34:45.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017750 con 0x7ff98000ac10 2021-03-09T10:34:45.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:45.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:45.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:45.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:45.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.741+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:45.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.745+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:45.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.745+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:45.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.745+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:45.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.745+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:45.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.745+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017890 con 0x7ff98000ac10 2021-03-09T10:34:45.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.745+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019870 con 0x7ff98000ac10 2021-03-09T10:34:45.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.745+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019a00 secure :-1 s=READY pgs=7167 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff9940aae90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:45.752 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:45.745+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24155 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:45.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.745+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940833b0 con 0x7ff98000ac10 2021-03-09T10:34:45.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.745+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940162e0 con 0x7ff98000ac10 2021-03-09T10:34:45.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.749+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff98000ac10 2021-03-09T10:34:45.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.749+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:45.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.749+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:45.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:45.749+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:50.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:50.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:50.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019a00 secure :-1 s=READY pgs=7167 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff9940aae90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:50.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019a00 secure :-1 s=READY pgs=7167 cs=0 l=1 rev1=1 rx=0x7ff99408e060 tx=0x7ff9940aae90).stop 2021-03-09T10:34:50.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019a00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:50.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019a00 unknown :-1 s=CLOSED pgs=7167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:50.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:50.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:50.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:50.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027160 con 0x7ff99c0adc20 2021-03-09T10:34:50.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0613a0 2021-03-09T10:34:50.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019870 con 0x7ff980083100 2021-03-09T10:34:50.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:50.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:50.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:50.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.744+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:50.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:50.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:50.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:50.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:50.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:50.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017a20 con 0x7ff980083100 2021-03-09T10:34:50.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017890 con 0x7ff980083100 2021-03-09T10:34:50.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.748+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7177 cs=0 l=1 rev1=1 rx=0x7ff980043e80 tx=0x7ff99408de20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:50.758 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:50.748+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24194 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:50.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994094540 con 0x7ff980083100 2021-03-09T10:34:50.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:34:50.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.756+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:34:50.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.756+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:50.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.756+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:50.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:50.756+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:34:55.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:55.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:55.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.748+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7177 cs=0 l=1 rev1=1 rx=0x7ff980043e80 tx=0x7ff99408de20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:55.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7177 cs=0 l=1 rev1=1 rx=0x7ff980043e80 tx=0x7ff99408de20).stop 2021-03-09T10:34:55.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:55.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:55.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:55.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:55.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:55.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005ce0 con 0x7ff99c0adc20 2021-03-09T10:34:55.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0613a0 2021-03-09T10:34:55.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017890 con 0x7ff98000ac10 2021-03-09T10:34:55.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:34:55.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:55.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:34:55.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:55.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:55.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:55.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:55.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:55.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:55.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980018090 con 0x7ff98000ac10 2021-03-09T10:34:55.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.752+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017a20 con 0x7ff98000ac10 2021-03-09T10:34:55.764 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:34:55.756+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24228 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:34:55.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.756+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7191 cs=0 l=1 rev1=1 rx=0x7ff994069040 tx=0x7ff994009e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:55.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.756+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff98000ac10 2021-03-09T10:34:55.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.756+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940835b0 con 0x7ff98000ac10 2021-03-09T10:34:55.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff98000ac10 2021-03-09T10:34:55.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.760+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:55.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.764+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:55.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:34:55.764+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:00.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:00.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:00.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7191 cs=0 l=1 rev1=1 rx=0x7ff994069040 tx=0x7ff994009e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:00.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7191 cs=0 l=1 rev1=1 rx=0x7ff994069040 tx=0x7ff994009e60).stop 2021-03-09T10:35:00.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:00.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=CLOSED pgs=7191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:00.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:00.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:00.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:00.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0adc20 2021-03-09T10:35:00.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017a30 con 0x7ff99c0613a0 2021-03-09T10:35:00.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017b70 con 0x7ff980083100 2021-03-09T10:35:00.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:00.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:00.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:00.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:00.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.756+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:00.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.760+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:00.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.760+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:00.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.760+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:00.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.760+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:00.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.760+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017cb0 con 0x7ff980083100 2021-03-09T10:35:00.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.760+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800196e0 con 0x7ff980083100 2021-03-09T10:35:00.766 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:00.760+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24268 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:00.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.760+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7203 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c079700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:00.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c07e520 con 0x7ff980083100 2021-03-09T10:35:00.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07e520 con 0x7ff980083100 2021-03-09T10:35:00.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.768+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:35:00.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.768+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:00.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.768+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:00.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:00.768+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:03.145 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:35:03.148 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:35:05.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:05.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:05.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7203 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c079700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:05.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7203 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c079700).stop 2021-03-09T10:35:05.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:05.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:05.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:05.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:05.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:05.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057440 con 0x7ff99c0adc20 2021-03-09T10:35:05.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0613a0 2021-03-09T10:35:05.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800196a0 con 0x7ff98000ac10 2021-03-09T10:35:05.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:05.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:05.768 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:05.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:05.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.760+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:05.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.764+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:05.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.764+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:05.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.764+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:05.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.764+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:05.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.764+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019810 con 0x7ff98000ac10 2021-03-09T10:35:05.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.764+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017cb0 con 0x7ff98000ac10 2021-03-09T10:35:05.771 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:05.764+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24302 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:05.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.764+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7212 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:05.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.768+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04bd70 con 0x7ff98000ac10 2021-03-09T10:35:05.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.768+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03da70 con 0x7ff98000ac10 2021-03-09T10:35:05.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.772+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bd70 con 0x7ff98000ac10 2021-03-09T10:35:05.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.772+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:05.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.772+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:05.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:05.772+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:10.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:10.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:10.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7212 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:10.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7212 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c005110).stop 2021-03-09T10:35:10.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:10.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=CLOSED pgs=7212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:10.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:10.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:10.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:10.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800199a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:10.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:10.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c170 con 0x7ff99c0adc20 2021-03-09T10:35:10.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800196a0 con 0x7ff99c0613a0 2021-03-09T10:35:10.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:35:10.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:10.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:10.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.764+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800199a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:10.773 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:10.768+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24342 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:10.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.768+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:10.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.768+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:10.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.768+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:10.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.768+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:10.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.768+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980057440 con 0x7ff980083100 2021-03-09T10:35:10.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.768+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980018090 con 0x7ff980083100 2021-03-09T10:35:10.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.768+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800199a0 secure :-1 s=READY pgs=7224 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940368f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:10.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.772+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:35:10.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.776+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:35:10.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.776+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:35:10.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.776+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:10.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.776+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:10.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:10.776+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:15.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800199a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:15.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800199a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:15.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800199a0 secure :-1 s=READY pgs=7224 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940368f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:15.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800199a0 secure :-1 s=READY pgs=7224 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940368f0).stop 2021-03-09T10:35:15.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800199a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:15.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800199a0 unknown :-1 s=CLOSED pgs=7224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:15.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:15.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:15.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:15.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800196a0 con 0x7ff99c0adc20 2021-03-09T10:35:15.776 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c170 con 0x7ff99c0613a0 2021-03-09T10:35:15.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff98000ac10 2021-03-09T10:35:15.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:15.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:15.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:15.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:15.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.768+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:15.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.772+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:15.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.772+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:15.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.772+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:15.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.772+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:15.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.772+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022ff0 con 0x7ff98000ac10 2021-03-09T10:35:15.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.772+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019810 con 0x7ff98000ac10 2021-03-09T10:35:15.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.772+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7232 cs=0 l=1 rev1=1 rx=0x7ff98c02d050 tx=0x7ff98c003ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:15.781 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:15.772+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24376 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:15.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.776+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff98000ac10 2021-03-09T10:35:15.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.776+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a9db0 con 0x7ff98000ac10 2021-03-09T10:35:15.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c09fde0 con 0x7ff98000ac10 2021-03-09T10:35:15.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.780+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:15.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.780+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:15.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:15.780+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:20.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:20.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:20.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7232 cs=0 l=1 rev1=1 rx=0x7ff98c02d050 tx=0x7ff98c003ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:20.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7232 cs=0 l=1 rev1=1 rx=0x7ff98c02d050 tx=0x7ff98c003ef0).stop 2021-03-09T10:35:20.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:20.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=CLOSED pgs=7232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:20.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:20.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:20.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:20.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff99c0adc20 2021-03-09T10:35:20.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019680 con 0x7ff99c0613a0 2021-03-09T10:35:20.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019810 con 0x7ff980083100 2021-03-09T10:35:20.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:20.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:20.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:20.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.772+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:20.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.776+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:20.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.776+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:20.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.776+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:20.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.776+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:20.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.776+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:20.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.776+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019980 con 0x7ff980083100 2021-03-09T10:35:20.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.776+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980057440 con 0x7ff980083100 2021-03-09T10:35:20.784 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:20.776+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24416 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:20.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.776+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7241 cs=0 l=1 rev1=1 rx=0x7ff99400d560 tx=0x7ff99400d480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:20.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.784+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408de90 con 0x7ff980083100 2021-03-09T10:35:20.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.784+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400d5d0 con 0x7ff980083100 2021-03-09T10:35:20.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.784+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:35:20.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.784+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:20.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.788+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:20.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:20.788+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7241 cs=0 l=1 rev1=1 rx=0x7ff99400d560 tx=0x7ff99400d480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7241 cs=0 l=1 rev1=1 rx=0x7ff99400d560 tx=0x7ff99400d480).stop 2021-03-09T10:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:25.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:25.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019680 con 0x7ff99c0adc20 2021-03-09T10:35:25.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff99c0613a0 2021-03-09T10:35:25.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019810 con 0x7ff98000ac10 2021-03-09T10:35:25.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:25.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:25.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:25.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.776+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:25.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.780+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:25.787 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:25.780+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24450 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:25.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.784+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:25.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.784+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:25.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.784+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:25.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.784+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:25.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.784+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019b10 con 0x7ff98000ac10 2021-03-09T10:35:25.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.784+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019980 con 0x7ff98000ac10 2021-03-09T10:35:25.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.784+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7259 cs=0 l=1 rev1=1 rx=0x7ff994081710 tx=0x7ff9940841f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:25.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.784+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940833b0 con 0x7ff98000ac10 2021-03-09T10:35:25.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994064090 con 0x7ff98000ac10 2021-03-09T10:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff98000ac10 2021-03-09T10:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.788+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:25.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.792+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:25.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:25.792+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:30.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:30.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:30.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7259 cs=0 l=1 rev1=1 rx=0x7ff994081710 tx=0x7ff9940841f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:30.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7259 cs=0 l=1 rev1=1 rx=0x7ff994081710 tx=0x7ff9940841f0).stop 2021-03-09T10:35:30.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:30.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=CLOSED pgs=7259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:30.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:30.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:30.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:30.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff99c0adc20 2021-03-09T10:35:30.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019680 con 0x7ff99c0613a0 2021-03-09T10:35:30.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019950 con 0x7ff980083100 2021-03-09T10:35:30.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:30.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:30.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:30.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:30.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:30.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:30.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:30.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:30.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:30.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980016fd0 con 0x7ff980083100 2021-03-09T10:35:30.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:35:30.799 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:30.784+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24489 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:30.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.784+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7264 cs=0 l=1 rev1=1 rx=0x7ff98000ee90 tx=0x7ff98c079780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:30.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0840e0 con 0x7ff980083100 2021-03-09T10:35:30.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0840e0 con 0x7ff980083100 2021-03-09T10:35:30.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.792+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:35:30.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.792+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:30.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.792+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:30.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:30.792+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:33.191 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:35:33.194 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:35:35.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:35.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:35.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7264 cs=0 l=1 rev1=1 rx=0x7ff98000ee90 tx=0x7ff98c079780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:35.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7264 cs=0 l=1 rev1=1 rx=0x7ff98000ee90 tx=0x7ff98c079780).stop 2021-03-09T10:35:35.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:35.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:35.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:35.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:35.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:35.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017160 con 0x7ff99c0adc20 2021-03-09T10:35:35.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff99c0613a0 2021-03-09T10:35:35.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cf60 con 0x7ff98000ac10 2021-03-09T10:35:35.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:35.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:35.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:35.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:35.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:35.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:35.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:35.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:35.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:35.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022ff0 con 0x7ff98000ac10 2021-03-09T10:35:35.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019b10 con 0x7ff98000ac10 2021-03-09T10:35:35.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.788+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7273 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99400d1c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:35.797 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:35.788+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24523 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:35.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.792+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409df20 con 0x7ff98000ac10 2021-03-09T10:35:35.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.792+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098120 con 0x7ff98000ac10 2021-03-09T10:35:35.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.796+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff98000ac10 2021-03-09T10:35:35.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.796+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:35.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.796+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:35.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:35.796+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:40.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:40.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:40.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7273 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99400d1c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:40.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7273 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99400d1c0).stop 2021-03-09T10:35:40.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:40.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=CLOSED pgs=7273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:40.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:40.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:40.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:40.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff99c0adc20 2021-03-09T10:35:40.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cf60 con 0x7ff99c0613a0 2021-03-09T10:35:40.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017160 con 0x7ff980083100 2021-03-09T10:35:40.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:40.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:40.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:40.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:40.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:40.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:40.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:40.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:40.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:40.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019680 con 0x7ff980083100 2021-03-09T10:35:40.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:35:40.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.792+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7285 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994085f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:40.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.796+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400c150 con 0x7ff980083100 2021-03-09T10:35:40.802 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:40.796+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24561 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:40.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.796+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098120 con 0x7ff980083100 2021-03-09T10:35:40.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.800+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:35:40.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.800+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:40.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.800+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:40.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:40.800+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:45.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:35:45.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:45.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7285 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994085f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:45.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7285 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994085f10).stop 2021-03-09T10:35:45.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:45.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:45.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:45.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:45.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:45.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093a10 con 0x7ff99c0adc20 2021-03-09T10:35:45.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cf60 con 0x7ff99c0613a0 2021-03-09T10:35:45.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff98000ac10 2021-03-09T10:35:45.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:45.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:45.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:45.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:45.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.796+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:45.806 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:45.800+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24598 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:45.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.800+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:45.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.800+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:45.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.800+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:45.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.800+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:45.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.800+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022ff0 con 0x7ff98000ac10 2021-03-09T10:35:45.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.800+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980016fd0 con 0x7ff98000ac10 2021-03-09T10:35:45.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.800+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7301 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c099950).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:45.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04bd70 con 0x7ff98000ac10 2021-03-09T10:35:45.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c099db0 con 0x7ff98000ac10 2021-03-09T10:35:45.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98000ac10 2021-03-09T10:35:45.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.812+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:45.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.812+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:45.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:45.812+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:50.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:50.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:50.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7301 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c099950).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:50.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7301 cs=0 l=1 rev1=1 rx=0x7ff98c079380 tx=0x7ff98c099950).stop 2021-03-09T10:35:50.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:50.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=CLOSED pgs=7301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:50.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:50.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:50.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:50.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff99c0adc20 2021-03-09T10:35:50.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093dc0 con 0x7ff99c0613a0 2021-03-09T10:35:50.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093f00 con 0x7ff980083100 2021-03-09T10:35:50.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:50.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:50.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:50.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:50.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:50.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:50.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:50.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:50.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:50.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092890 con 0x7ff980083100 2021-03-09T10:35:50.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.800+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019680 con 0x7ff980083100 2021-03-09T10:35:50.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.804+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7305 cs=0 l=1 rev1=1 rx=0x7ff99409bb80 tx=0x7ff9940011b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:50.811 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:50.804+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24634 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:50.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400d230 con 0x7ff980083100 2021-03-09T10:35:50.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940835b0 con 0x7ff980083100 2021-03-09T10:35:50.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff980083100 2021-03-09T10:35:50.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.808+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:50.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.812+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:50.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:50.812+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:35:55.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:55.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:55.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7305 cs=0 l=1 rev1=1 rx=0x7ff99409bb80 tx=0x7ff9940011b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:55.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7305 cs=0 l=1 rev1=1 rx=0x7ff99409bb80 tx=0x7ff9940011b0).stop 2021-03-09T10:35:55.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:55.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:55.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:55.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:55.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:55.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff99c0adc20 2021-03-09T10:35:55.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c530 con 0x7ff99c0613a0 2021-03-09T10:35:55.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046a90 con 0x7ff98000ac10 2021-03-09T10:35:55.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:35:55.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:35:55.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:55.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:55.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:55.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:55.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:55.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:55.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:55.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980046bd0 con 0x7ff98000ac10 2021-03-09T10:35:55.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.804+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092890 con 0x7ff98000ac10 2021-03-09T10:35:55.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.808+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7317 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff9940ac1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:55.815 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:35:55.808+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24672 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:35:55.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408dd40 con 0x7ff98000ac10 2021-03-09T10:35:55.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.808+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940835b0 con 0x7ff98000ac10 2021-03-09T10:35:55.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:35:55.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.812+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:55.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.816+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:55.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:35:55.816+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:00.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:00.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:00.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7317 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff9940ac1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:00.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 secure :-1 s=READY pgs=7317 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff9940ac1e0).stop 2021-03-09T10:36:00.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008c8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:00.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008c8e0 unknown :-1 s=CLOSED pgs=7317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:00.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:00.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:00.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:00.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c530 con 0x7ff99c0adc20 2021-03-09T10:36:00.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff99c0613a0 2021-03-09T10:36:00.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800395d0 con 0x7ff980083100 2021-03-09T10:36:00.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:00.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:00.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:00.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:00.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:00.818 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:00.812+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24707 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:00.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:00.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:00.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:00.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:00.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980046a40 con 0x7ff980083100 2021-03-09T10:36:00.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.812+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:36:00.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.816+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7333 cs=0 l=1 rev1=1 rx=0x7ff98c045ce0 tx=0x7ff98c022ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:00.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c045b20 con 0x7ff980083100 2021-03-09T10:36:00.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c045b20 con 0x7ff980083100 2021-03-09T10:36:00.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:36:00.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.824+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:00.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.824+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:00.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:00.824+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:03.345 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:36:03.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:36:05.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:05.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:05.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7333 cs=0 l=1 rev1=1 rx=0x7ff98c045ce0 tx=0x7ff98c022ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:05.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 secure :-1 s=READY pgs=7333 cs=0 l=1 rev1=1 rx=0x7ff98c045ce0 tx=0x7ff98c022ef0).stop 2021-03-09T10:36:05.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:05.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232d0 unknown :-1 s=CLOSED pgs=7333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:05.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:05.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:05.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:05.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800406d0 con 0x7ff99c0adc20 2021-03-09T10:36:05.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002ffc0 con 0x7ff99c0613a0 2021-03-09T10:36:05.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff98000ac10 2021-03-09T10:36:05.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:05.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:05.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:05.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:05.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:05.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:05.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:05.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:05.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:05.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022ff0 con 0x7ff98000ac10 2021-03-09T10:36:05.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.816+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980046bd0 con 0x7ff98000ac10 2021-03-09T10:36:05.825 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:05.816+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24746 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:05.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.820+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800067b0 secure :-1 s=READY pgs=7339 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400bed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:05.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405d090 con 0x7ff98000ac10 2021-03-09T10:36:05.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.820+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405d090 con 0x7ff98000ac10 2021-03-09T10:36:05.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.824+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:36:05.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.824+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:05.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.828+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:05.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:05.828+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:10.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:10.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:10.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800067b0 secure :-1 s=READY pgs=7339 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400bed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:10.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800067b0 secure :-1 s=READY pgs=7339 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400bed0).stop 2021-03-09T10:36:10.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800067b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:10.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800067b0 unknown :-1 s=CLOSED pgs=7339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:10.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:10.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:10.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:10.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800131e0 con 0x7ff99c0adc20 2021-03-09T10:36:10.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980040870 con 0x7ff99c0613a0 2021-03-09T10:36:10.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c530 con 0x7ff980083100 2021-03-09T10:36:10.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:10.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:10.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:10.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:10.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:10.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:10.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:10.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:10.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:10.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096990 con 0x7ff980083100 2021-03-09T10:36:10.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.819+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980046a40 con 0x7ff980083100 2021-03-09T10:36:10.834 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:10.819+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24782 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:10.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.823+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232a0 secure :-1 s=READY pgs=7350 cs=0 l=1 rev1=1 rx=0x7ff98000c280 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:10.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.823+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c084e80 con 0x7ff980083100 2021-03-09T10:36:10.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.823+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:36:10.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0229e0 con 0x7ff980083100 2021-03-09T10:36:10.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.827+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:10.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.831+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:10.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:10.831+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:15.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:15.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:15.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232a0 secure :-1 s=READY pgs=7350 cs=0 l=1 rev1=1 rx=0x7ff98000c280 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:15.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232a0 secure :-1 s=READY pgs=7350 cs=0 l=1 rev1=1 rx=0x7ff98000c280 tx=0x7ff98c003e80).stop 2021-03-09T10:36:15.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800232a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:15.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800232a0 unknown :-1 s=CLOSED pgs=7350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:15.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:15.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:15.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:15.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980030500 con 0x7ff99c0adc20 2021-03-09T10:36:15.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c530 con 0x7ff99c0613a0 2021-03-09T10:36:15.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022b40 con 0x7ff98000ac10 2021-03-09T10:36:15.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:15.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:15.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:15.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:15.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:15.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:15.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:15.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:15.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:15.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002ffc0 con 0x7ff98000ac10 2021-03-09T10:36:15.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.823+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022ff0 con 0x7ff98000ac10 2021-03-09T10:36:15.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096b20 secure :-1 s=READY pgs=7361 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99402a180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:15.834 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:15.827+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24820 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:15.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff98000ac10 2021-03-09T10:36:15.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940835b0 con 0x7ff98000ac10 2021-03-09T10:36:15.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.835+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:36:15.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.835+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:15.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.835+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:15.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:15.835+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:20.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:20.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:20.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096b20 secure :-1 s=READY pgs=7361 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99402a180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:20.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096b20 secure :-1 s=READY pgs=7361 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99402a180).stop 2021-03-09T10:36:20.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:20.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096b20 unknown :-1 s=CLOSED pgs=7361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:20.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:20.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:20.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:20.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:20.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:20.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019e90 con 0x7ff99c0adc20 2021-03-09T10:36:20.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004db40 con 0x7ff99c0613a0 2021-03-09T10:36:20.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a7e0 con 0x7ff980083100 2021-03-09T10:36:20.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:20.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:20.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:20.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:20.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:20.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:20.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:20.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022ff0 con 0x7ff980083100 2021-03-09T10:36:20.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.827+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096990 con 0x7ff980083100 2021-03-09T10:36:20.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.831+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006570 secure :-1 s=READY pgs=7369 cs=0 l=1 rev1=1 rx=0x7ff98c09f740 tx=0x7ff98c09f5f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:20.839 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:20.831+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24856 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:20.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.835+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:36:20.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.835+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980083100 2021-03-09T10:36:20.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.835+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:36:20.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.835+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:20.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.839+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:20.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:20.839+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:25.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980006570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:25.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980006570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:25.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006570 secure :-1 s=READY pgs=7369 cs=0 l=1 rev1=1 rx=0x7ff98c09f740 tx=0x7ff98c09f5f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:25.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006570 secure :-1 s=READY pgs=7369 cs=0 l=1 rev1=1 rx=0x7ff98c09f740 tx=0x7ff98c09f5f0).stop 2021-03-09T10:36:25.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980006570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:25.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006570 unknown :-1 s=CLOSED pgs=7369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:25.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:25.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:25.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:25.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019e90 con 0x7ff99c0adc20 2021-03-09T10:36:25.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a7e0 con 0x7ff99c0613a0 2021-03-09T10:36:25.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ffe0 con 0x7ff98000ac10 2021-03-09T10:36:25.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:25.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:25.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:25.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.831+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:25.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.835+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:25.842 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:25.835+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24894 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:25.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.835+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:25.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.835+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:25.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.835+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:25.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.835+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:25.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.835+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004daf0 con 0x7ff98000ac10 2021-03-09T10:36:25.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.835+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002ffc0 con 0x7ff98000ac10 2021-03-09T10:36:25.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.839+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 secure :-1 s=READY pgs=7387 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff9940114e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:25.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.839+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409bd50 con 0x7ff98000ac10 2021-03-09T10:36:25.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.839+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940835b0 con 0x7ff98000ac10 2021-03-09T10:36:25.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:36:25.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.843+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:25.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.847+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:25.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:25.847+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:30.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:36:30.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:30.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 secure :-1 s=READY pgs=7387 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff9940114e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:30.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 secure :-1 s=READY pgs=7387 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff9940114e0).stop 2021-03-09T10:36:30.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:30.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 unknown :-1 s=CLOSED pgs=7387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:30.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:30.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:30.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:30.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a7e0 con 0x7ff99c0adc20 2021-03-09T10:36:30.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090180 con 0x7ff99c0613a0 2021-03-09T10:36:30.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:30.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046240 con 0x7ff980083100 2021-03-09T10:36:30.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:30.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:30.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:30.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.835+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:30.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.839+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:30.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.839+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:30.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.839+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:30.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.839+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:30.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.839+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002ffc0 con 0x7ff980083100 2021-03-09T10:36:30.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.839+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004daf0 con 0x7ff980083100 2021-03-09T10:36:30.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.839+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800064d0 secure :-1 s=READY pgs=7391 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff9940aabb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:30.849 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:30.839+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24929 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:30.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.839+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009a20 con 0x7ff980083100 2021-03-09T10:36:30.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.839+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940aad00 con 0x7ff980083100 2021-03-09T10:36:30.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940aad00 con 0x7ff980083100 2021-03-09T10:36:30.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.843+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:30.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.843+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:30.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:30.843+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:33.567 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:36:33.570 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:36:35.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:35.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:35.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800064d0 secure :-1 s=READY pgs=7391 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff9940aabb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:35.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800064d0 secure :-1 s=READY pgs=7391 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff9940aabb0).stop 2021-03-09T10:36:35.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800064d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:35.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800064d0 unknown :-1 s=CLOSED pgs=7391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:35.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:35.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:35.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:35.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800075e0 con 0x7ff99c0adc20 2021-03-09T10:36:35.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980040d50 con 0x7ff99c0613a0 2021-03-09T10:36:35.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046240 con 0x7ff98000ac10 2021-03-09T10:36:35.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:35.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:35.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:35.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:35.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:35.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:35.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:35.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:35.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:35.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013380 con 0x7ff98000ac10 2021-03-09T10:36:35.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022ff0 con 0x7ff98000ac10 2021-03-09T10:36:35.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.843+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 secure :-1 s=READY pgs=7401 cs=0 l=1 rev1=1 rx=0x7ff980019a40 tx=0x7ff98c00bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:35.852 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:35.843+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24968 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:35.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.847+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c09f170 con 0x7ff98000ac10 2021-03-09T10:36:35.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.847+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c09f170 con 0x7ff98000ac10 2021-03-09T10:36:35.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98000ac10 2021-03-09T10:36:35.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.851+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:35.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.851+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:35.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:35.851+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:40.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:40.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:40.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 secure :-1 s=READY pgs=7401 cs=0 l=1 rev1=1 rx=0x7ff980019a40 tx=0x7ff98c00bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:40.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 secure :-1 s=READY pgs=7401 cs=0 l=1 rev1=1 rx=0x7ff980019a40 tx=0x7ff98c00bfd0).stop 2021-03-09T10:36:40.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:40.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023220 unknown :-1 s=CLOSED pgs=7401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:40.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:40.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:40.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:40.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046240 con 0x7ff99c0adc20 2021-03-09T10:36:40.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0613a0 2021-03-09T10:36:40.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020ae0 con 0x7ff980083100 2021-03-09T10:36:40.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:40.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:40.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:40.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:40.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.847+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:40.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:40.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:40.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:40.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:40.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020c20 con 0x7ff980083100 2021-03-09T10:36:40.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002ffc0 con 0x7ff980083100 2021-03-09T10:36:40.858 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:40.851+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25003 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:40.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006450 secure :-1 s=READY pgs=7414 cs=0 l=1 rev1=1 rx=0x7ff98000f590 tx=0x7ff99400bf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:40.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994084360 con 0x7ff980083100 2021-03-09T10:36:40.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994084180 con 0x7ff980083100 2021-03-09T10:36:40.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:36:40.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.859+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:40.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.859+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:40.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:40.859+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:45.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980006450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:45.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980006450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:45.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006450 secure :-1 s=READY pgs=7414 cs=0 l=1 rev1=1 rx=0x7ff98000f590 tx=0x7ff99400bf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:45.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006450 secure :-1 s=READY pgs=7414 cs=0 l=1 rev1=1 rx=0x7ff98000f590 tx=0x7ff99400bf40).stop 2021-03-09T10:36:45.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980006450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:45.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980006450 unknown :-1 s=CLOSED pgs=7414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:45.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:45.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:45.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:45.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020ae0 con 0x7ff99c0adc20 2021-03-09T10:36:45.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001a020 con 0x7ff99c0613a0 2021-03-09T10:36:45.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980058830 con 0x7ff98000ac10 2021-03-09T10:36:45.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:45.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:45.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:45.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:45.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.851+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:45.861 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:45.855+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25042 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:45.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.855+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:45.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.855+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:45.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.855+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:45.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.855+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:45.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.855+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002ffc0 con 0x7ff98000ac10 2021-03-09T10:36:45.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.855+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013380 con 0x7ff98000ac10 2021-03-09T10:36:45.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.855+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 secure :-1 s=READY pgs=7428 cs=0 l=1 rev1=1 rx=0x7ff98004b5f0 tx=0x7ff98c08e180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:45.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff98000ac10 2021-03-09T10:36:45.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98000ac10 2021-03-09T10:36:45.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c09f170 con 0x7ff98000ac10 2021-03-09T10:36:45.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.867+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:45.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.867+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:45.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:45.867+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:50.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980022ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:50.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980022ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:50.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 secure :-1 s=READY pgs=7428 cs=0 l=1 rev1=1 rx=0x7ff98004b5f0 tx=0x7ff98c08e180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:50.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 secure :-1 s=READY pgs=7428 cs=0 l=1 rev1=1 rx=0x7ff98004b5f0 tx=0x7ff98c08e180).stop 2021-03-09T10:36:50.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980022ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:50.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 unknown :-1 s=CLOSED pgs=7428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:50.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800131e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:50.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:50.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:50.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027190 con 0x7ff99c0adc20 2021-03-09T10:36:50.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005010 con 0x7ff99c0613a0 2021-03-09T10:36:50.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005150 con 0x7ff980083100 2021-03-09T10:36:50.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:50.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:50.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:50.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:50.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800131e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:50.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:50.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:50.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:50.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:50.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800063e0 con 0x7ff980083100 2021-03-09T10:36:50.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020c20 con 0x7ff980083100 2021-03-09T10:36:50.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800131e0 secure :-1 s=READY pgs=7433 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff994086e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:50.868 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:50.859+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25078 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:50.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980083100 2021-03-09T10:36:50.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994094780 con 0x7ff980083100 2021-03-09T10:36:50.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:36:50.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.867+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:50.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.867+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:50.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:50.867+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:36:55.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800131e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:55.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800131e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:55.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800131e0 secure :-1 s=READY pgs=7433 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff994086e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:55.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800131e0 secure :-1 s=READY pgs=7433 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff994086e50).stop 2021-03-09T10:36:55.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800131e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:55.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800131e0 unknown :-1 s=CLOSED pgs=7433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:55.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:55.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:55.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:55.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800234e0 con 0x7ff99c0adc20 2021-03-09T10:36:55.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020c20 con 0x7ff99c0613a0 2021-03-09T10:36:55.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020d60 con 0x7ff98000ac10 2021-03-09T10:36:55.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:36:55.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:55.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:36:55.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:55.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:55.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:55.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:55.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:55.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:55.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980006570 con 0x7ff98000ac10 2021-03-09T10:36:55.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800063e0 con 0x7ff98000ac10 2021-03-09T10:36:55.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 secure :-1 s=READY pgs=7445 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff9940948e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:55.873 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:36:55.863+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25116 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:36:55.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081340 con 0x7ff98000ac10 2021-03-09T10:36:55.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:36:55.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:36:55.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.871+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:55.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.871+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:55.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:36:55.871+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:00.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980022ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:00.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980022ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:00.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 secure :-1 s=READY pgs=7445 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff9940948e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:00.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 secure :-1 s=READY pgs=7445 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff9940948e0).stop 2021-03-09T10:37:00.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980022ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:00.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 unknown :-1 s=CLOSED pgs=7445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:00.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:00.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:00.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:00.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800068a0 con 0x7ff99c0adc20 2021-03-09T10:37:00.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013530 con 0x7ff99c0613a0 2021-03-09T10:37:00.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b2b0 con 0x7ff980083100 2021-03-09T10:37:00.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:00.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:00.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:00.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:00.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:00.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:00.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:00.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:00.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:00.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004b3f0 con 0x7ff980083100 2021-03-09T10:37:00.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980006570 con 0x7ff980083100 2021-03-09T10:37:00.877 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:00.871+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25152 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:00.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 secure :-1 s=READY pgs=7459 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99400dc00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:00.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994014b10 con 0x7ff980083100 2021-03-09T10:37:00.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994014b10 con 0x7ff980083100 2021-03-09T10:37:00.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.879+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:37:00.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.879+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:00.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.879+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:00.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:00.879+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:03.611 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:37:03.614 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:37:05.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:05.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:05.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 secure :-1 s=READY pgs=7459 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99400dc00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:05.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 secure :-1 s=READY pgs=7459 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99400dc00).stop 2021-03-09T10:37:05.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:05.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 unknown :-1 s=CLOSED pgs=7459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:05.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:05.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:05.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:05.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c970 con 0x7ff99c0adc20 2021-03-09T10:37:05.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800234e0 con 0x7ff99c0613a0 2021-03-09T10:37:05.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:05.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055380 con 0x7ff98000ac10 2021-03-09T10:37:05.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:05.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:05.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:05.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:05.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:05.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:05.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:05.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:05.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800554f0 con 0x7ff98000ac10 2021-03-09T10:37:05.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.871+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002ffc0 con 0x7ff98000ac10 2021-03-09T10:37:05.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 secure :-1 s=READY pgs=7466 cs=0 l=1 rev1=1 rx=0x7ff98c08e240 tx=0x7ff98c00ba80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:05.885 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:05.875+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25190 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:05.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08f130 con 0x7ff98000ac10 2021-03-09T10:37:05.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff98000ac10 2021-03-09T10:37:05.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c07dd90 con 0x7ff98000ac10 2021-03-09T10:37:05.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.883+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:05.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.883+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:05.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:05.883+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:10.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980022ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:10.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980022ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:10.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 secure :-1 s=READY pgs=7466 cs=0 l=1 rev1=1 rx=0x7ff98c08e240 tx=0x7ff98c00ba80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:10.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 secure :-1 s=READY pgs=7466 cs=0 l=1 rev1=1 rx=0x7ff98c08e240 tx=0x7ff98c00ba80).stop 2021-03-09T10:37:10.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980022ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:10.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980022ff0 unknown :-1 s=CLOSED pgs=7466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:10.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:10.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:10.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:10.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013530 con 0x7ff99c0adc20 2021-03-09T10:37:10.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008a040 con 0x7ff99c0613a0 2021-03-09T10:37:10.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057580 con 0x7ff980083100 2021-03-09T10:37:10.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:10.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:10.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:10.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:10.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:10.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:10.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:10.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:10.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.879+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:10.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055680 con 0x7ff980083100 2021-03-09T10:37:10.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.879+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800554f0 con 0x7ff980083100 2021-03-09T10:37:10.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.879+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 secure :-1 s=READY pgs=7477 cs=0 l=1 rev1=1 rx=0x7ff98c023040 tx=0x7ff98c006690).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:10.888 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:10.879+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25226 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:10.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08f130 con 0x7ff980083100 2021-03-09T10:37:10.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0450a0 con 0x7ff980083100 2021-03-09T10:37:10.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00b330 con 0x7ff980083100 2021-03-09T10:37:10.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.887+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:10.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.887+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:10.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:10.887+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:15.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:15.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:15.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 secure :-1 s=READY pgs=7477 cs=0 l=1 rev1=1 rx=0x7ff98c023040 tx=0x7ff98c006690).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:15.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 secure :-1 s=READY pgs=7477 cs=0 l=1 rev1=1 rx=0x7ff98c023040 tx=0x7ff98c006690).stop 2021-03-09T10:37:15.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:15.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 unknown :-1 s=CLOSED pgs=7477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:15.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:15.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:15.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:15.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057580 con 0x7ff99c0adc20 2021-03-09T10:37:15.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022ff0 con 0x7ff99c0613a0 2021-03-09T10:37:15.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023130 con 0x7ff98000ac10 2021-03-09T10:37:15.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:15.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:15.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:15.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:15.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:15.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:15.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:15.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:15.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:15.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980023270 con 0x7ff98000ac10 2021-03-09T10:37:15.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004b3f0 con 0x7ff98000ac10 2021-03-09T10:37:15.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.883+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055180 secure :-1 s=READY pgs=7487 cs=0 l=1 rev1=1 rx=0x7ff99400c010 tx=0x7ff994081670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:15.891 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:15.883+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25264 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:15.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940835b0 con 0x7ff98000ac10 2021-03-09T10:37:15.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff98000ac10 2021-03-09T10:37:15.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:37:15.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.891+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:15.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.891+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:15.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:15.891+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:20.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:20.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:20.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055180 secure :-1 s=READY pgs=7487 cs=0 l=1 rev1=1 rx=0x7ff99400c010 tx=0x7ff994081670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:20.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055180 secure :-1 s=READY pgs=7487 cs=0 l=1 rev1=1 rx=0x7ff99400c010 tx=0x7ff994081670).stop 2021-03-09T10:37:20.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:20.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055180 unknown :-1 s=CLOSED pgs=7487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:20.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:20.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:20.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:20.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013530 con 0x7ff99c0adc20 2021-03-09T10:37:20.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980089ea0 con 0x7ff99c0613a0 2021-03-09T10:37:20.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980089fe0 con 0x7ff980083100 2021-03-09T10:37:20.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:20.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:20.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:20.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:20.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:20.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:20.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:20.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:20.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000eff0 con 0x7ff980083100 2021-03-09T10:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.887+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980023270 con 0x7ff980083100 2021-03-09T10:37:20.897 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:20.887+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25300 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:20.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.891+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 secure :-1 s=READY pgs=7500 cs=0 l=1 rev1=1 rx=0x7ff994068d50 tx=0x7ff994068a10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:20.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980083100 2021-03-09T10:37:20.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.891+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980083100 2021-03-09T10:37:20.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.899+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994068dc0 con 0x7ff980083100 2021-03-09T10:37:20.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.899+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:20.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.899+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:20.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:20.899+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:25.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:25.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:25.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 secure :-1 s=READY pgs=7500 cs=0 l=1 rev1=1 rx=0x7ff994068d50 tx=0x7ff994068a10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:25.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 secure :-1 s=READY pgs=7500 cs=0 l=1 rev1=1 rx=0x7ff994068d50 tx=0x7ff994068a10).stop 2021-03-09T10:37:25.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:25.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013040 unknown :-1 s=CLOSED pgs=7500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:25.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:25.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:25.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:25.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d4c0 con 0x7ff99c0adc20 2021-03-09T10:37:25.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055330 con 0x7ff99c0613a0 2021-03-09T10:37:25.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055470 con 0x7ff98000ac10 2021-03-09T10:37:25.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:25.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:25.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:25.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:25.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:25.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:25.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:25.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:25.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:25.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980023270 con 0x7ff98000ac10 2021-03-09T10:37:25.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.890+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000eff0 con 0x7ff98000ac10 2021-03-09T10:37:25.901 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:25.894+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25334 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:25.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 secure :-1 s=READY pgs=7512 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994034460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:25.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff98000ac10 2021-03-09T10:37:25.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff98000ac10 2021-03-09T10:37:25.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994068dc0 con 0x7ff98000ac10 2021-03-09T10:37:25.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.902+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:25.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.902+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:25.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:25.902+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:30.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:30.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:30.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 secure :-1 s=READY pgs=7512 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994034460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:30.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 secure :-1 s=READY pgs=7512 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994034460).stop 2021-03-09T10:37:30.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980054e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:30.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 unknown :-1 s=CLOSED pgs=7512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:30.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:30.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:30.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:30.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013500 con 0x7ff99c0adc20 2021-03-09T10:37:30.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046fd0 con 0x7ff99c0613a0 2021-03-09T10:37:30.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:30.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047110 con 0x7ff980083100 2021-03-09T10:37:30.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:30.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:30.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:30.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:30.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:30.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:30.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:30.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:30.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055330 con 0x7ff980083100 2021-03-09T10:37:30.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.894+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055680 con 0x7ff980083100 2021-03-09T10:37:30.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.898+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013010 secure :-1 s=READY pgs=7520 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c08e1a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:30.907 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:30.898+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25374 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:30.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.898+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0872a0 con 0x7ff980083100 2021-03-09T10:37:30.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.898+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:37:30.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff980083100 2021-03-09T10:37:30.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.906+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:30.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.906+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:30.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:30.906+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:33.667 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:37:33.670 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:37:35.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:35.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:35.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013010 secure :-1 s=READY pgs=7520 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c08e1a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:35.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013010 secure :-1 s=READY pgs=7520 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c08e1a0).stop 2021-03-09T10:37:35.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980013010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:35.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980013010 unknown :-1 s=CLOSED pgs=7520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:35.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:35.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:35.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:35.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:35.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:35.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019630 con 0x7ff99c0adc20 2021-03-09T10:37:35.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019770 con 0x7ff99c0613a0 2021-03-09T10:37:35.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054e40 con 0x7ff98000ac10 2021-03-09T10:37:35.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:35.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:35.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:35.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:35.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:35.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:35.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:35.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054f80 con 0x7ff98000ac10 2021-03-09T10:37:35.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980023270 con 0x7ff98000ac10 2021-03-09T10:37:35.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7529 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff9940816e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:35.912 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:35.902+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25411 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:35.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:37:35.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994094130 con 0x7ff98000ac10 2021-03-09T10:37:35.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:37:35.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.910+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:35.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.910+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:35.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:35.910+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:40.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:40.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:40.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7529 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff9940816e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:40.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7529 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff9940816e0).stop 2021-03-09T10:37:40.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:40.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=CLOSED pgs=7529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:40.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:40.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:40.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:40.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054e40 con 0x7ff99c0adc20 2021-03-09T10:37:40.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a780 con 0x7ff99c0613a0 2021-03-09T10:37:40.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800134b0 con 0x7ff980083100 2021-03-09T10:37:40.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:40.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:40.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:40.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:40.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:40.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.906+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:40.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.906+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:40.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.906+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:40.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.906+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:40.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.906+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096470 con 0x7ff980083100 2021-03-09T10:37:40.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.906+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054f80 con 0x7ff980083100 2021-03-09T10:37:40.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.906+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012fc0 secure :-1 s=READY pgs=7542 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99400d9f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:40.916 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:40.906+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25448 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:40.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff980083100 2021-03-09T10:37:40.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff980083100 2021-03-09T10:37:40.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:37:40.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.914+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:40.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.914+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:40.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:40.914+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:45.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:45.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:45.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012fc0 secure :-1 s=READY pgs=7542 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99400d9f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:45.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012fc0 secure :-1 s=READY pgs=7542 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff99400d9f0).stop 2021-03-09T10:37:45.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:45.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012fc0 unknown :-1 s=CLOSED pgs=7542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:45.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:45.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:45.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:45.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800928d0 con 0x7ff99c0adc20 2021-03-09T10:37:45.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a780 con 0x7ff99c0613a0 2021-03-09T10:37:45.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054e40 con 0x7ff98000ac10 2021-03-09T10:37:45.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:45.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:45.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:45.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:45.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:45.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:45.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:45.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:45.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:45.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054fb0 con 0x7ff98000ac10 2021-03-09T10:37:45.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096470 con 0x7ff98000ac10 2021-03-09T10:37:45.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7551 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff99409dfe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:45.922 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:45.914+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25485 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:45.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405d090 con 0x7ff98000ac10 2021-03-09T10:37:45.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940942f0 con 0x7ff98000ac10 2021-03-09T10:37:45.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:37:45.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.918+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:45.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.918+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:45.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:45.918+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:50.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:50.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:50.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7551 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff99409dfe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:50.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7551 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff99409dfe0).stop 2021-03-09T10:37:50.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:50.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=CLOSED pgs=7551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:50.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:50.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:50.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:50.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:50.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:50.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:50.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cc90 con 0x7ff99c0adc20 2021-03-09T10:37:50.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c3f0 con 0x7ff99c0613a0 2021-03-09T10:37:50.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800928d0 con 0x7ff980083100 2021-03-09T10:37:50.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:50.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:50.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:50.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:50.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:50.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:50.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013410 con 0x7ff980083100 2021-03-09T10:37:50.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054fb0 con 0x7ff980083100 2021-03-09T10:37:50.926 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:50.918+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25522 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:50.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.918+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012f20 secure :-1 s=READY pgs=7561 cs=0 l=1 rev1=1 rx=0x7ff99405edd0 tx=0x7ff994036930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:50.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994098300 con 0x7ff980083100 2021-03-09T10:37:50.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098120 con 0x7ff980083100 2021-03-09T10:37:50.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:37:50.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.926+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:50.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.930+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:50.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:50.930+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:37:55.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:55.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:55.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012f20 secure :-1 s=READY pgs=7561 cs=0 l=1 rev1=1 rx=0x7ff99405edd0 tx=0x7ff994036930).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:55.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012f20 secure :-1 s=READY pgs=7561 cs=0 l=1 rev1=1 rx=0x7ff99405edd0 tx=0x7ff994036930).stop 2021-03-09T10:37:55.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:55.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012f20 unknown :-1 s=CLOSED pgs=7561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:55.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:55.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:55.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:55.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c3f0 con 0x7ff99c0adc20 2021-03-09T10:37:55.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cc90 con 0x7ff99c0613a0 2021-03-09T10:37:55.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800928d0 con 0x7ff98000ac10 2021-03-09T10:37:55.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:37:55.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:55.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:37:55.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:55.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:55.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:55.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:55.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:55.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:55.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096420 con 0x7ff98000ac10 2021-03-09T10:37:55.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013410 con 0x7ff98000ac10 2021-03-09T10:37:55.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7573 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff99400dad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:55.931 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:37:55.922+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25559 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:37:55.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940851f0 con 0x7ff98000ac10 2021-03-09T10:37:55.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff98000ac10 2021-03-09T10:37:55.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:37:55.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.930+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:55.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.930+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:55.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:37:55.930+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:00.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:00.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:00.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7573 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff99400dad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:00.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7573 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff99400dad0).stop 2021-03-09T10:38:00.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:00.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=CLOSED pgs=7573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:00.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:00.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:00.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:00.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cc90 con 0x7ff99c0adc20 2021-03-09T10:38:00.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c3f0 con 0x7ff99c0613a0 2021-03-09T10:38:00.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003a8d0 con 0x7ff980083100 2021-03-09T10:38:00.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:00.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:00.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:00.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:00.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:00.934 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:00.926+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25596 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:00.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:00.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:00.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:00.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:00.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800133e0 con 0x7ff980083100 2021-03-09T10:38:00.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096420 con 0x7ff980083100 2021-03-09T10:38:00.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800965b0 secure :-1 s=READY pgs=7588 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994067280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:00.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069350 con 0x7ff980083100 2021-03-09T10:38:00.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940851f0 con 0x7ff980083100 2021-03-09T10:38:00.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:38:00.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.934+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:00.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.934+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:00.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:00.934+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:03.732 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:38:03.735 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:38:05.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:05.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:05.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800965b0 secure :-1 s=READY pgs=7588 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994067280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:05.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800965b0 secure :-1 s=READY pgs=7588 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994067280).stop 2021-03-09T10:38:05.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800965b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:05.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800965b0 unknown :-1 s=CLOSED pgs=7588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:05.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:05.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:05.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:05.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c3f0 con 0x7ff99c0adc20 2021-03-09T10:38:05.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057de0 con 0x7ff99c0613a0 2021-03-09T10:38:05.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cc90 con 0x7ff98000ac10 2021-03-09T10:38:05.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:05.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:05.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:05.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:05.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:05.938 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:05.930+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25633 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:05.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:05.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:05.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:05.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:05.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096aa0 con 0x7ff98000ac10 2021-03-09T10:38:05.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.930+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055330 con 0x7ff98000ac10 2021-03-09T10:38:05.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.934+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7597 cs=0 l=1 rev1=1 rx=0x7ff98c084ad0 tx=0x7ff98c08e080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:05.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:38:05.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:38:05.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a0130 con 0x7ff98000ac10 2021-03-09T10:38:05.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.942+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:05.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.942+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:05.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:05.942+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:10.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:10.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:10.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7597 cs=0 l=1 rev1=1 rx=0x7ff98c084ad0 tx=0x7ff98c08e080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:10.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7597 cs=0 l=1 rev1=1 rx=0x7ff98c084ad0 tx=0x7ff98c08e080).stop 2021-03-09T10:38:10.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:10.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=CLOSED pgs=7597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:10.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:10.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:10.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:10.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c3f0 con 0x7ff99c0adc20 2021-03-09T10:38:10.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800098f0 con 0x7ff99c0613a0 2021-03-09T10:38:10.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044bf0 con 0x7ff980083100 2021-03-09T10:38:10.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:10.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:10.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:10.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:10.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:10.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:10.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:10.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:10.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:10.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055330 con 0x7ff980083100 2021-03-09T10:38:10.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096aa0 con 0x7ff980083100 2021-03-09T10:38:10.946 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:10.934+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25670 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:10.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7605 cs=0 l=1 rev1=1 rx=0x7ff98c0a2e60 tx=0x7ff98c0223c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:10.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00b120 con 0x7ff980083100 2021-03-09T10:38:10.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00b280 con 0x7ff980083100 2021-03-09T10:38:10.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00b280 con 0x7ff980083100 2021-03-09T10:38:10.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.942+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:10.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.942+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:10.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:10.942+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:15.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:38:15.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:15.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7605 cs=0 l=1 rev1=1 rx=0x7ff98c0a2e60 tx=0x7ff98c0223c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:15.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7605 cs=0 l=1 rev1=1 rx=0x7ff98c0a2e60 tx=0x7ff98c0223c0).stop 2021-03-09T10:38:15.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:15.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=7605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:15.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:15.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:15.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:15.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c3f0 con 0x7ff99c0adc20 2021-03-09T10:38:15.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044bf0 con 0x7ff99c0613a0 2021-03-09T10:38:15.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a110 con 0x7ff98000ac10 2021-03-09T10:38:15.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:15.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:15.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:15.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:15.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:15.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:15.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:15.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:15.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:15.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800098f0 con 0x7ff98000ac10 2021-03-09T10:38:15.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055330 con 0x7ff98000ac10 2021-03-09T10:38:15.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.942+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7615 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c003f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:15.950 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:15.942+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25707 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:15.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00b120 con 0x7ff98000ac10 2021-03-09T10:38:15.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00b280 con 0x7ff98000ac10 2021-03-09T10:38:15.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff98000ac10 2021-03-09T10:38:15.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.946+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:15.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.946+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:15.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:15.946+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:20.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:20.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:20.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7615 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c003f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:20.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 secure :-1 s=READY pgs=7615 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c003f70).stop 2021-03-09T10:38:20.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800554c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:20.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800554c0 unknown :-1 s=CLOSED pgs=7615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:20.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:20.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:20.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:20.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980044bf0 con 0x7ff99c0adc20 2021-03-09T10:38:20.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800237b0 con 0x7ff99c0613a0 2021-03-09T10:38:20.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cc90 con 0x7ff980083100 2021-03-09T10:38:20.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:20.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:20.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:20.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:20.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:20.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:20.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:20.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:20.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:20.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004f000 con 0x7ff980083100 2021-03-09T10:38:20.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.942+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800098f0 con 0x7ff980083100 2021-03-09T10:38:20.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.946+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7626 cs=0 l=1 rev1=1 rx=0x7ff98c07a170 tx=0x7ff98c0043b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:20.957 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:20.946+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25744 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:20.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08d860 con 0x7ff980083100 2021-03-09T10:38:20.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08d680 con 0x7ff980083100 2021-03-09T10:38:20.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c08dcc0 con 0x7ff980083100 2021-03-09T10:38:20.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.954+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:20.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.954+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:20.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:20.954+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:25.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:25.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:25.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7626 cs=0 l=1 rev1=1 rx=0x7ff98c07a170 tx=0x7ff98c0043b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:25.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7626 cs=0 l=1 rev1=1 rx=0x7ff98c07a170 tx=0x7ff98c0043b0).stop 2021-03-09T10:38:25.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:25.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=7626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:25.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:25.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:25.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:25.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800557f0 con 0x7ff99c0adc20 2021-03-09T10:38:25.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009940 con 0x7ff99c0613a0 2021-03-09T10:38:25.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f190 con 0x7ff98000ac10 2021-03-09T10:38:25.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:25.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:25.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:25.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:25.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:25.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:25.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:25.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:25.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:25.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980029f50 con 0x7ff98000ac10 2021-03-09T10:38:25.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004f000 con 0x7ff98000ac10 2021-03-09T10:38:25.966 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:25.950+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25781 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:25.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.950+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 secure :-1 s=READY pgs=7641 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c008e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:25.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04c140 con 0x7ff98000ac10 2021-03-09T10:38:25.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04c140 con 0x7ff98000ac10 2021-03-09T10:38:25.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:38:25.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.958+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:25.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.962+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:25.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:25.962+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:30.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:30.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:30.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 secure :-1 s=READY pgs=7641 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c008e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:30.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 secure :-1 s=READY pgs=7641 cs=0 l=1 rev1=1 rx=0x7ff98c00b840 tx=0x7ff98c008e60).stop 2021-03-09T10:38:30.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:30.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 unknown :-1 s=CLOSED pgs=7641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:30.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:30.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:30.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:30.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a0e0 con 0x7ff99c0adc20 2021-03-09T10:38:30.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980085590 con 0x7ff99c0613a0 2021-03-09T10:38:30.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:30.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800856d0 con 0x7ff980083100 2021-03-09T10:38:30.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:30.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:30.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:30.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:30.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:30.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:30.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:30.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:30.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047230 con 0x7ff980083100 2021-03-09T10:38:30.965 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:30.954+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25817 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:30.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800133e0 con 0x7ff980083100 2021-03-09T10:38:30.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.958+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7651 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994069480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:30.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069600 con 0x7ff980083100 2021-03-09T10:38:30.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff980083100 2021-03-09T10:38:30.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.966+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:38:30.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.966+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:30.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.966+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:30.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:30.966+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:33.789 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:38:33.792 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:38:35.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:35.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:35.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7651 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994069480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:35.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7651 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994069480).stop 2021-03-09T10:38:35.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:35.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=7651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:35.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:35.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:35.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:35.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:35.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047560 con 0x7ff99c0adc20 2021-03-09T10:38:35.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:35.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980028650 con 0x7ff99c0613a0 2021-03-09T10:38:35.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a0e0 con 0x7ff98000ac10 2021-03-09T10:38:35.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:35.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:35.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:35.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:35.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:35.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:35.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:35.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800557f0 con 0x7ff98000ac10 2021-03-09T10:38:35.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047230 con 0x7ff98000ac10 2021-03-09T10:38:35.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.958+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 secure :-1 s=READY pgs=7657 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff9940814f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:35.971 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:35.962+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25855 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:35.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009a80 con 0x7ff98000ac10 2021-03-09T10:38:35.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff98000ac10 2021-03-09T10:38:35.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.966+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994014da0 con 0x7ff98000ac10 2021-03-09T10:38:35.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.966+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:35.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.966+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:35.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:35.966+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:40.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:40.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:40.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 secure :-1 s=READY pgs=7657 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff9940814f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:40.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 secure :-1 s=READY pgs=7657 cs=0 l=1 rev1=1 rx=0x7ff994083540 tx=0x7ff9940814f0).stop 2021-03-09T10:38:40.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:40.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055300 unknown :-1 s=CLOSED pgs=7657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:40.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:40.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:40.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:40.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:40.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980028630 con 0x7ff99c0adc20 2021-03-09T10:38:40.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:40.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a0e0 con 0x7ff99c0613a0 2021-03-09T10:38:40.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011270 con 0x7ff980083100 2021-03-09T10:38:40.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:40.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:40.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.962+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:40.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.966+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:40.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.966+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:40.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.966+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:40.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.966+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:40.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.966+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff980083100 2021-03-09T10:38:40.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.966+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980029f50 con 0x7ff980083100 2021-03-09T10:38:40.975 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:40.966+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25892 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:40.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.970+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7674 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00ba80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:40.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.974+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c008cc0 con 0x7ff980083100 2021-03-09T10:38:40.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.974+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a9120 con 0x7ff980083100 2021-03-09T10:38:40.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.974+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:38:40.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.974+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:40.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.974+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:40.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:40.974+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:45.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:45.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:45.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7674 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00ba80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:45.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7674 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00ba80).stop 2021-03-09T10:38:45.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:45.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=7674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:45.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:45.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:45.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:45.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f340 con 0x7ff99c0adc20 2021-03-09T10:38:45.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980028630 con 0x7ff99c0613a0 2021-03-09T10:38:45.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800063c0 con 0x7ff98000ac10 2021-03-09T10:38:45.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:45.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:45.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:45.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:45.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:45.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:45.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:45.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:45.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:45.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022b20 con 0x7ff98000ac10 2021-03-09T10:38:45.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800557f0 con 0x7ff98000ac10 2021-03-09T10:38:45.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029c10 secure :-1 s=READY pgs=7679 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff9940702c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:45.981 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:45.969+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25929 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:45.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.969+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff98000ac10 2021-03-09T10:38:45.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994086e50 con 0x7ff98000ac10 2021-03-09T10:38:45.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99408d550 con 0x7ff98000ac10 2021-03-09T10:38:45.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.977+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:45.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.977+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:45.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:45.977+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:49.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 msgr2=0x7ff980049a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:38:49.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 msgr2=0x7ff980049a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:49.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 0x7ff980049a30 secure :-1 s=READY pgs=5484 cs=0 l=1 rev1=1 rx=0x7ff984049060 tx=0x7ff984000d00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:49.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 0x7ff980049a30 secure :-1 s=READY pgs=5484 cs=0 l=1 rev1=1 rx=0x7ff984049060 tx=0x7ff984000d00).stop 2021-03-09T10:38:49.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 msgr2=0x7ff980049a30 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:38:49.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff9937fe700 4 mgrc ms_handle_reset ms_handle_reset con 0x7ff980051540 2021-03-09T10:38:49.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff9937fe700 4 mgrc reconnect Terminating session with v2:172.21.15.52:6808/14433 2021-03-09T10:38:49.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 msgr2=0x7ff980049a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:49.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 0x7ff980049a30 unknown :-1 s=CLOSED pgs=5484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:49.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff9937fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:49.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:49.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:49.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:49.897+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7290 cs=0 l=1 rev1=1 rx=0x7ff98407b4a0 tx=0x7ff98407d070).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:50.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:50.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.969+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:50.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029c10 secure :-1 s=READY pgs=7679 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff9940702c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:50.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.969+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029c10 secure :-1 s=READY pgs=7679 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff9940702c0).stop 2021-03-09T10:38:50.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:50.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029c10 unknown :-1 s=CLOSED pgs=7679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:50.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:50.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:50.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:50.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005ef0 con 0x7ff99c0adc20 2021-03-09T10:38:50.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004f340 con 0x7ff99c0613a0 2021-03-09T10:38:50.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a100 con 0x7ff980051540 2021-03-09T10:38:50.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:50.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:50.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:50.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:50.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:50.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:50.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:50.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:50.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:50.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980010f10 con 0x7ff980051540 2021-03-09T10:38:50.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022b20 con 0x7ff980051540 2021-03-09T10:38:50.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 secure :-1 s=READY pgs=7689 cs=0 l=1 rev1=1 rx=0x7ff994000fa0 tx=0x7ff994094540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:50.986 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:50.977+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25966 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:50.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99404a020 con 0x7ff980051540 2021-03-09T10:38:50.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095b10 con 0x7ff980051540 2021-03-09T10:38:50.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980051540 2021-03-09T10:38:50.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.981+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:50.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.981+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:50.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:50.981+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:38:55.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980049a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:55.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980049a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:55.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 secure :-1 s=READY pgs=7689 cs=0 l=1 rev1=1 rx=0x7ff994000fa0 tx=0x7ff994094540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:55.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 secure :-1 s=READY pgs=7689 cs=0 l=1 rev1=1 rx=0x7ff994000fa0 tx=0x7ff994094540).stop 2021-03-09T10:38:55.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980049a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:55.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049a30 unknown :-1 s=CLOSED pgs=7689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:55.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:55.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:55.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:55.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022de0 con 0x7ff99c0adc20 2021-03-09T10:38:55.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029f00 con 0x7ff99c0613a0 2021-03-09T10:38:55.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a040 con 0x7ff98000ac10 2021-03-09T10:38:55.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:55.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:38:55.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:55.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:38:55.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:55.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:55.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:55.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:55.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:55.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020610 con 0x7ff98000ac10 2021-03-09T10:38:55.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:38:55.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.977+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029a10 secure :-1 s=READY pgs=7701 cs=0 l=1 rev1=1 rx=0x7ff98c0992e0 tx=0x7ff98c017f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:55.996 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:38:55.981+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26003 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:38:55.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff98000ac10 2021-03-09T10:38:55.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98000ac10 2021-03-09T10:38:55.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:38:55.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.985+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:55.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.989+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:55.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:38:55.989+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:00.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:00.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:00.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029a10 secure :-1 s=READY pgs=7701 cs=0 l=1 rev1=1 rx=0x7ff98c0992e0 tx=0x7ff98c017f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:00.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029a10 secure :-1 s=READY pgs=7701 cs=0 l=1 rev1=1 rx=0x7ff98c0992e0 tx=0x7ff98c017f30).stop 2021-03-09T10:39:00.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029a10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:00.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029a10 unknown :-1 s=CLOSED pgs=7701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:00.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:00.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:00.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:00.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ed80 con 0x7ff99c0adc20 2021-03-09T10:39:00.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff99c0613a0 2021-03-09T10:39:00.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800260d0 con 0x7ff980051540 2021-03-09T10:39:00.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:00.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:00.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:00.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:00.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:00.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:00.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:00.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:00.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:00.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004bce0 con 0x7ff980051540 2021-03-09T10:39:00.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980010f10 con 0x7ff980051540 2021-03-09T10:39:00.996 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:00.985+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26040 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:00.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800499f0 secure :-1 s=READY pgs=7712 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99400bf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:00.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994029af0 con 0x7ff980051540 2021-03-09T10:39:00.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994001680 con 0x7ff980051540 2021-03-09T10:39:01.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:39:01.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.993+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:01.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.993+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:01.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:00.993+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:03.833 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:39:03.836 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:39:05.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800499f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:05.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800499f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:05.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800499f0 secure :-1 s=READY pgs=7712 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99400bf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:05.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800499f0 secure :-1 s=READY pgs=7712 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99400bf40).stop 2021-03-09T10:39:05.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800499f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:05.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800499f0 unknown :-1 s=CLOSED pgs=7712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:05.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:05.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:05.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:05.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022e30 con 0x7ff99c0adc20 2021-03-09T10:39:05.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ed60 con 0x7ff99c0613a0 2021-03-09T10:39:05.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092cc0 con 0x7ff98000ac10 2021-03-09T10:39:05.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:05.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:05.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:05.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:05.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.989+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:05.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:05.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:06.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:06.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:06.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020470 con 0x7ff98000ac10 2021-03-09T10:39:06.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004bce0 con 0x7ff98000ac10 2021-03-09T10:39:06.002 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:05.993+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26077 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:06.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7725 cs=0 l=1 rev1=1 rx=0x7ff98008be40 tx=0x7ff994060310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:06.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994014120 con 0x7ff98000ac10 2021-03-09T10:39:06.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:05.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994094130 con 0x7ff98000ac10 2021-03-09T10:39:06.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:06.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940b11c0 con 0x7ff98000ac10 2021-03-09T10:39:06.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:06.001+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:06.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:06.001+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:06.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:06.001+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:10.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:10.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:11.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7725 cs=0 l=1 rev1=1 rx=0x7ff98008be40 tx=0x7ff994060310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:11.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7725 cs=0 l=1 rev1=1 rx=0x7ff98008be40 tx=0x7ff994060310).stop 2021-03-09T10:39:11.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:11.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=CLOSED pgs=7725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:11.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:11.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:11.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:11.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004c200 con 0x7ff99c0adc20 2021-03-09T10:39:11.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020a70 con 0x7ff99c0613a0 2021-03-09T10:39:11.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022e30 con 0x7ff980051540 2021-03-09T10:39:11.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:11.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:11.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:11.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:11.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:11.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:10.997+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:11.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:11.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:11.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:11.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020470 con 0x7ff980051540 2021-03-09T10:39:11.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004bce0 con 0x7ff980051540 2021-03-09T10:39:11.006 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:11.001+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26114 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:11.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049800 secure :-1 s=READY pgs=7733 cs=0 l=1 rev1=1 rx=0x7ff994094510 tx=0x7ff994098370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:11.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940984e0 con 0x7ff980051540 2021-03-09T10:39:11.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940986f0 con 0x7ff980051540 2021-03-09T10:39:11.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994029cb0 con 0x7ff980051540 2021-03-09T10:39:11.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.005+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:11.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.005+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:11.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:11.005+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:16.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980049800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:16.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980049800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:16.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049800 secure :-1 s=READY pgs=7733 cs=0 l=1 rev1=1 rx=0x7ff994094510 tx=0x7ff994098370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:16.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049800 secure :-1 s=READY pgs=7733 cs=0 l=1 rev1=1 rx=0x7ff994094510 tx=0x7ff994098370).stop 2021-03-09T10:39:16.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980049800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:16.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980049800 unknown :-1 s=CLOSED pgs=7733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:16.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:16.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:16.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:16.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022e30 con 0x7ff99c0adc20 2021-03-09T10:39:16.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020a40 con 0x7ff99c0613a0 2021-03-09T10:39:16.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020470 con 0x7ff98000ac10 2021-03-09T10:39:16.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:16.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:16.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:16.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:16.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.001+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:16.008 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:16.005+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26151 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:16.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:16.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.005+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:16.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:16.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.005+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:16.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980029e50 con 0x7ff98000ac10 2021-03-09T10:39:16.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020610 con 0x7ff98000ac10 2021-03-09T10:39:16.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.005+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7747 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c0056c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:16.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:39:16.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:39:16.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.013+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:39:16.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.013+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:16.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.013+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:16.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:16.013+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:21.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:21.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:21.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7747 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c0056c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:21.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7747 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c0056c0).stop 2021-03-09T10:39:21.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:21.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=CLOSED pgs=7747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:21.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:21.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:21.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:21.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800291d0 con 0x7ff99c0adc20 2021-03-09T10:39:21.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022e30 con 0x7ff99c0613a0 2021-03-09T10:39:21.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800920e0 con 0x7ff980051540 2021-03-09T10:39:21.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:21.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:21.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:21.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:21.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.005+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:21.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:21.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:21.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:21.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:21.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020a40 con 0x7ff980051540 2021-03-09T10:39:21.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004bce0 con 0x7ff980051540 2021-03-09T10:39:21.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7754 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994070230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:21.015 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:21.009+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26186 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:21.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980051540 2021-03-09T10:39:21.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407a180 con 0x7ff980051540 2021-03-09T10:39:21.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:39:21.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.017+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:21.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.017+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:21.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:21.017+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:26.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:26.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:26.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7754 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994070230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:26.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7754 cs=0 l=1 rev1=1 rx=0x7ff9940819d0 tx=0x7ff994070230).stop 2021-03-09T10:39:26.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:26.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:26.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:26.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:26.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:26.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018230 con 0x7ff99c0adc20 2021-03-09T10:39:26.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017150 con 0x7ff99c0613a0 2021-03-09T10:39:26.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022e30 con 0x7ff98000ac10 2021-03-09T10:39:26.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:26.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:26.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:26.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:26.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:26.018 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:26.013+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26225 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:26.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.013+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:26.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.013+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:26.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.013+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:26.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.013+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:26.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.013+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980029180 con 0x7ff98000ac10 2021-03-09T10:39:26.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.013+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020a40 con 0x7ff98000ac10 2021-03-09T10:39:26.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.017+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7770 cs=0 l=1 rev1=1 rx=0x7ff994020730 tx=0x7ff99400c640).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:26.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994014120 con 0x7ff98000ac10 2021-03-09T10:39:26.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.021+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994014120 con 0x7ff98000ac10 2021-03-09T10:39:26.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.021+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:39:26.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.021+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:26.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.025+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:26.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:26.025+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:31.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:31.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:31.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7770 cs=0 l=1 rev1=1 rx=0x7ff994020730 tx=0x7ff99400c640).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:31.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7770 cs=0 l=1 rev1=1 rx=0x7ff994020730 tx=0x7ff99400c640).stop 2021-03-09T10:39:31.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:31.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=CLOSED pgs=7770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:31.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:31.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:31.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:31.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017150 con 0x7ff99c0adc20 2021-03-09T10:39:31.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022e30 con 0x7ff99c0613a0 2021-03-09T10:39:31.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020a40 con 0x7ff980051540 2021-03-09T10:39:31.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:31.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:31.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:31.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:31.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.013+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:31.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:31.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.017+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:31.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:31.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.017+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:31.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d9f0 con 0x7ff980051540 2021-03-09T10:39:31.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980029180 con 0x7ff980051540 2021-03-09T10:39:31.025 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:31.017+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26261 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:31.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.021+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7780 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff994098120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:31.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.021+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99407e6a0 con 0x7ff980051540 2021-03-09T10:39:31.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.021+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99407e470 con 0x7ff980051540 2021-03-09T10:39:31.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:39:31.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.029+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:31.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.029+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:31.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:31.029+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:33.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:39:33.890 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:39:36.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:36.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:36.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.017+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7780 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff994098120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:36.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.017+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7780 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff994098120).stop 2021-03-09T10:39:36.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:36.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.017+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:36.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.017+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:36.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.021+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:36.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.021+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:36.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.021+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017150 con 0x7ff99c0adc20 2021-03-09T10:39:36.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.021+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020a40 con 0x7ff99c0613a0 2021-03-09T10:39:36.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.021+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003690 con 0x7ff98000ac10 2021-03-09T10:39:36.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.021+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:36.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.021+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:36.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.021+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:36.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.021+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:36.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.021+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:36.028 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:36.025+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26299 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:36.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:36.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:36.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:36.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:36.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022e10 con 0x7ff98000ac10 2021-03-09T10:39:36.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d9f0 con 0x7ff98000ac10 2021-03-09T10:39:36.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7791 cs=0 l=1 rev1=1 rx=0x7ff99400b0b0 tx=0x7ff9940700e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:36.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:39:36.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940aae20 con 0x7ff98000ac10 2021-03-09T10:39:36.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:39:36.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.033+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:36.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.033+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:36.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:36.033+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:41.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:41.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:41.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7791 cs=0 l=1 rev1=1 rx=0x7ff99400b0b0 tx=0x7ff9940700e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:41.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7791 cs=0 l=1 rev1=1 rx=0x7ff99400b0b0 tx=0x7ff9940700e0).stop 2021-03-09T10:39:41.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:41.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=CLOSED pgs=7791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:41.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:41.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:41.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:41.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093080 con 0x7ff99c0adc20 2021-03-09T10:39:41.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012fe0 con 0x7ff99c0613a0 2021-03-09T10:39:41.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cea0 con 0x7ff980051540 2021-03-09T10:39:41.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:41.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:41.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:41.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:41.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:41.033 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:41.029+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26334 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:41.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:41.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:41.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:41.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:41.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d6b0 con 0x7ff980051540 2021-03-09T10:39:41.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022e10 con 0x7ff980051540 2021-03-09T10:39:41.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7803 cs=0 l=1 rev1=1 rx=0x7ff99409c040 tx=0x7ff9940aed10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:41.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994068310 con 0x7ff980051540 2021-03-09T10:39:41.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994068130 con 0x7ff980051540 2021-03-09T10:39:41.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:39:41.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.037+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:41.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.037+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:41.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:41.037+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:46.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:46.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:46.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7803 cs=0 l=1 rev1=1 rx=0x7ff99409c040 tx=0x7ff9940aed10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:46.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7803 cs=0 l=1 rev1=1 rx=0x7ff99409c040 tx=0x7ff9940aed10).stop 2021-03-09T10:39:46.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:46.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:46.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:46.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:46.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:46.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017150 con 0x7ff99c0adc20 2021-03-09T10:39:46.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cea0 con 0x7ff99c0613a0 2021-03-09T10:39:46.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012fe0 con 0x7ff98000ac10 2021-03-09T10:39:46.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:46.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:46.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:46.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:46.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:46.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:46.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:46.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:46.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:46.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022e10 con 0x7ff98000ac10 2021-03-09T10:39:46.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020610 con 0x7ff98000ac10 2021-03-09T10:39:46.038 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:46.033+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26373 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:46.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7808 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c00bf60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:46.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c07d0f0 con 0x7ff98000ac10 2021-03-09T10:39:46.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.041+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07d0f0 con 0x7ff98000ac10 2021-03-09T10:39:46.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.041+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:39:46.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.041+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:46.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.041+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:46.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:46.041+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:51.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:51.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:51.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7808 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c00bf60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:51.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 secure :-1 s=READY pgs=7808 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c00bf60).stop 2021-03-09T10:39:51.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980029960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:51.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980029960 unknown :-1 s=CLOSED pgs=7808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:51.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:51.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:51.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:51.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cea0 con 0x7ff99c0adc20 2021-03-09T10:39:51.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017150 con 0x7ff99c0613a0 2021-03-09T10:39:51.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff980051540 2021-03-09T10:39:51.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:51.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:51.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:51.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:51.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.033+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:51.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.037+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:51.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.037+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:51.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.037+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:51.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.037+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:51.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.037+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff980051540 2021-03-09T10:39:51.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.037+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d6b0 con 0x7ff980051540 2021-03-09T10:39:51.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.037+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7817 cs=0 l=1 rev1=1 rx=0x7ff99403f040 tx=0x7ff99400c640).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:51.044 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:51.037+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26409 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:51.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99401ed80 con 0x7ff980051540 2021-03-09T10:39:51.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff980051540 2021-03-09T10:39:51.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.045+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:39:51.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.045+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:51.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.045+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:51.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:51.045+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:39:56.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:56.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:56.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7817 cs=0 l=1 rev1=1 rx=0x7ff99403f040 tx=0x7ff99400c640).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:56.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7817 cs=0 l=1 rev1=1 rx=0x7ff99403f040 tx=0x7ff99400c640).stop 2021-03-09T10:39:56.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:56.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:56.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:56.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:56.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:56.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cea0 con 0x7ff99c0adc20 2021-03-09T10:39:56.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017150 con 0x7ff99c0613a0 2021-03-09T10:39:56.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff98000ac10 2021-03-09T10:39:56.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:39:56.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:56.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:39:56.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:56.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.037+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:56.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.041+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:56.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.041+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:56.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.041+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:56.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.041+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:56.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.041+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022940 con 0x7ff98000ac10 2021-03-09T10:39:56.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.041+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022e10 con 0x7ff98000ac10 2021-03-09T10:39:56.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.041+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096a40 secure :-1 s=READY pgs=7830 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c0080a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:56.050 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:39:56.041+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26447 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:39:56.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.045+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98000ac10 2021-03-09T10:39:56.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.045+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:39:56.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.049+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98000ac10 2021-03-09T10:39:56.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.049+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:56.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.049+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:56.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:39:56.049+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:01.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:01.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:01.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096a40 secure :-1 s=READY pgs=7830 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c0080a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:01.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096a40 secure :-1 s=READY pgs=7830 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c0080a0).stop 2021-03-09T10:40:01.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:01.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096a40 unknown :-1 s=CLOSED pgs=7830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:01.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:01.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:01.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:01.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cea0 con 0x7ff99c0adc20 2021-03-09T10:40:01.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017150 con 0x7ff99c0613a0 2021-03-09T10:40:01.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:01.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff980051540 2021-03-09T10:40:01.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:01.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:01.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:01.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.044+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:01.055 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:01.048+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26483 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:01.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.048+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:01.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.048+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:01.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.048+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:01.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.048+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:01.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.048+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022e10 con 0x7ff980051540 2021-03-09T10:40:01.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.048+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022940 con 0x7ff980051540 2021-03-09T10:40:01.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.048+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7845 cs=0 l=1 rev1=1 rx=0x7ff98c022da0 tx=0x7ff98c052130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:01.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00b330 con 0x7ff980051540 2021-03-09T10:40:01.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T10:40:01.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00b330 con 0x7ff980051540 2021-03-09T10:40:01.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.056+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:01.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.056+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:01.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:01.056+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:03.935 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:40:03.938 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:40:06.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:06.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:06.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7845 cs=0 l=1 rev1=1 rx=0x7ff98c022da0 tx=0x7ff98c052130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:06.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7845 cs=0 l=1 rev1=1 rx=0x7ff98c022da0 tx=0x7ff98c052130).stop 2021-03-09T10:40:06.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:06.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:06.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:06.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:06.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:06.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008cea0 con 0x7ff99c0adc20 2021-03-09T10:40:06.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009470 con 0x7ff99c0613a0 2021-03-09T10:40:06.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012e80 con 0x7ff98000ac10 2021-03-09T10:40:06.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:06.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:06.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:06.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.048+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:06.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:06.058 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:06.052+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26521 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:06.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:06.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:06.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:06.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:06.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027970 con 0x7ff98000ac10 2021-03-09T10:40:06.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:40:06.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 secure :-1 s=READY pgs=7856 cs=0 l=1 rev1=1 rx=0x7ff980009de0 tx=0x7ff9940368f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:06.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409f940 con 0x7ff98000ac10 2021-03-09T10:40:06.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99402a180 con 0x7ff98000ac10 2021-03-09T10:40:06.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:40:06.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.060+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:06.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.064+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:06.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:06.064+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:11.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98001eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:11.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98001eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:11.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 secure :-1 s=READY pgs=7856 cs=0 l=1 rev1=1 rx=0x7ff980009de0 tx=0x7ff9940368f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:11.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 secure :-1 s=READY pgs=7856 cs=0 l=1 rev1=1 rx=0x7ff980009de0 tx=0x7ff9940368f0).stop 2021-03-09T10:40:11.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98001eb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:11.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 unknown :-1 s=CLOSED pgs=7856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:11.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:11.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:11.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:11.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046b60 con 0x7ff99c0adc20 2021-03-09T10:40:11.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800198f0 con 0x7ff99c0613a0 2021-03-09T10:40:11.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017150 con 0x7ff980051540 2021-03-09T10:40:11.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:11.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:11.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:11.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:11.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.052+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:11.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:11.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:11.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:11.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:11.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff980051540 2021-03-09T10:40:11.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.056+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027970 con 0x7ff980051540 2021-03-09T10:40:11.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.056+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7861 cs=0 l=1 rev1=1 rx=0x7ff994060010 tx=0x7ff99402a180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:11.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:40:11.068 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:11.056+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26557 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:11.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.056+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:40:11.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940700d0 con 0x7ff980051540 2021-03-09T10:40:11.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.060+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:11.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.060+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:11.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:11.060+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:16.063 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:16.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:16.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7861 cs=0 l=1 rev1=1 rx=0x7ff994060010 tx=0x7ff99402a180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:16.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7861 cs=0 l=1 rev1=1 rx=0x7ff994060010 tx=0x7ff99402a180).stop 2021-03-09T10:40:16.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:16.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:16.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:16.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:16.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:16.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020de0 con 0x7ff99c0adc20 2021-03-09T10:40:16.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017150 con 0x7ff99c0613a0 2021-03-09T10:40:16.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:16.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:16.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:16.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:16.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:16.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff98000ac10 2021-03-09T10:40:16.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:16.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:16.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:16.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:16.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980009450 con 0x7ff98000ac10 2021-03-09T10:40:16.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022e10 con 0x7ff98000ac10 2021-03-09T10:40:16.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.060+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 secure :-1 s=READY pgs=7871 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c0a2e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:16.075 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:16.060+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26596 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:16.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.064+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:40:16.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.064+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c09f170 con 0x7ff98000ac10 2021-03-09T10:40:16.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c049040 con 0x7ff98000ac10 2021-03-09T10:40:16.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.068+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:16.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.068+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:16.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:16.068+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:21.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98001eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:21.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98001eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:21.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 secure :-1 s=READY pgs=7871 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c0a2e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:21.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 secure :-1 s=READY pgs=7871 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c0a2e20).stop 2021-03-09T10:40:21.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98001eb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:21.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 unknown :-1 s=CLOSED pgs=7871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:21.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:21.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:21.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:21.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000aab0 con 0x7ff99c0adc20 2021-03-09T10:40:21.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff99c0613a0 2021-03-09T10:40:21.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017de0 con 0x7ff980051540 2021-03-09T10:40:21.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:21.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:21.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:21.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:21.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:21.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:21.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:21.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:21.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:21.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980046b10 con 0x7ff980051540 2021-03-09T10:40:21.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.064+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff980051540 2021-03-09T10:40:21.074 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:21.068+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26631 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:21.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.068+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7882 cs=0 l=1 rev1=1 rx=0x7ff99408d310 tx=0x7ff99400bf30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:21.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940aec80 con 0x7ff980051540 2021-03-09T10:40:21.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940aec80 con 0x7ff980051540 2021-03-09T10:40:21.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99409bde0 con 0x7ff980051540 2021-03-09T10:40:21.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.076+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:21.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.076+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:21.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:21.076+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:26.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:26.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:26.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7882 cs=0 l=1 rev1=1 rx=0x7ff99408d310 tx=0x7ff99400bf30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:26.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7882 cs=0 l=1 rev1=1 rx=0x7ff99408d310 tx=0x7ff99400bf30).stop 2021-03-09T10:40:26.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:26.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:26.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:26.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:26.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:26.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023740 con 0x7ff99c0adc20 2021-03-09T10:40:26.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff99c0613a0 2021-03-09T10:40:26.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000aab0 con 0x7ff98000ac10 2021-03-09T10:40:26.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:26.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:26.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:26.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:26.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:26.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:26.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:26.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:26.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:26.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980027970 con 0x7ff98000ac10 2021-03-09T10:40:26.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.068+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980046b10 con 0x7ff98000ac10 2021-03-09T10:40:26.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.072+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014760 secure :-1 s=READY pgs=7893 cs=0 l=1 rev1=1 rx=0x7ff9800405c0 tx=0x7ff9940aa920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:26.083 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:26.072+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26670 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:26.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940014c0 con 0x7ff98000ac10 2021-03-09T10:40:26.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940abc40 con 0x7ff98000ac10 2021-03-09T10:40:26.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff98000ac10 2021-03-09T10:40:26.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.076+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:26.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.080+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:26.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:26.080+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:31.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:31.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:31.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014760 secure :-1 s=READY pgs=7893 cs=0 l=1 rev1=1 rx=0x7ff9800405c0 tx=0x7ff9940aa920).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:31.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014760 secure :-1 s=READY pgs=7893 cs=0 l=1 rev1=1 rx=0x7ff9800405c0 tx=0x7ff9940aa920).stop 2021-03-09T10:40:31.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:31.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014760 unknown :-1 s=CLOSED pgs=7893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:31.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:31.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:31.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:31.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:31.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff99c0adc20 2021-03-09T10:40:31.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:31.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000aab0 con 0x7ff99c0613a0 2021-03-09T10:40:31.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014df0 con 0x7ff980051540 2021-03-09T10:40:31.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:31.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:31.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.072+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:31.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.076+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:31.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.076+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:31.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.076+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:31.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.076+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:31.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.076+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022e10 con 0x7ff980051540 2021-03-09T10:40:31.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.076+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980009450 con 0x7ff980051540 2021-03-09T10:40:31.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.076+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7903 cs=0 l=1 rev1=1 rx=0x7ff980089d70 tx=0x7ff98c00bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:31.085 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:31.076+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26706 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:31.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:40:31.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2c00 con 0x7ff980051540 2021-03-09T10:40:31.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c049050 con 0x7ff980051540 2021-03-09T10:40:31.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.084+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:31.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.084+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:31.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:31.084+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:33.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:40:34.001 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:40:36.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:36.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:36.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7903 cs=0 l=1 rev1=1 rx=0x7ff980089d70 tx=0x7ff98c00bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:36.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7903 cs=0 l=1 rev1=1 rx=0x7ff980089d70 tx=0x7ff98c00bfd0).stop 2021-03-09T10:40:36.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:36.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:36.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:36.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:36.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:36.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000aab0 con 0x7ff99c0adc20 2021-03-09T10:40:36.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff99c0613a0 2021-03-09T10:40:36.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004dc40 con 0x7ff98000ac10 2021-03-09T10:40:36.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:36.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:36.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:36.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.076+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:36.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.080+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:36.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.080+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:36.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.080+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:36.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.080+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:36.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.080+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:36.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.080+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980046b10 con 0x7ff98000ac10 2021-03-09T10:40:36.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.080+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022e10 con 0x7ff98000ac10 2021-03-09T10:40:36.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.080+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 secure :-1 s=READY pgs=7914 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c0a9c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:36.089 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:36.084+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26745 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:36.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c022340 con 0x7ff98000ac10 2021-03-09T10:40:36.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:40:36.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c004c00 con 0x7ff98000ac10 2021-03-09T10:40:36.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.088+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:36.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.088+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:36.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:36.088+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:41.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98001eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:41.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98001eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:41.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 secure :-1 s=READY pgs=7914 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c0a9c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:41.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 secure :-1 s=READY pgs=7914 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c0a9c60).stop 2021-03-09T10:40:41.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98001eb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:41.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98001eb20 unknown :-1 s=CLOSED pgs=7914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:41.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:41.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:41.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:41.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800363d0 con 0x7ff99c0adc20 2021-03-09T10:40:41.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff99c0613a0 2021-03-09T10:40:41.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000aab0 con 0x7ff980051540 2021-03-09T10:40:41.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:41.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:41.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:41.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:41.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.084+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:41.092 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:41.088+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26779 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:41.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:41.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:41.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:41.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:41.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017490 con 0x7ff980051540 2021-03-09T10:40:41.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980027970 con 0x7ff980051540 2021-03-09T10:40:41.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7931 cs=0 l=1 rev1=1 rx=0x7ff9940946b0 tx=0x7ff99400dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:41.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff980051540 2021-03-09T10:40:41.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff980051540 2021-03-09T10:40:41.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.096+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940abc40 con 0x7ff980051540 2021-03-09T10:40:41.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.096+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:41.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.096+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:41.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:41.096+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:46.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:46.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:46.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7931 cs=0 l=1 rev1=1 rx=0x7ff9940946b0 tx=0x7ff99400dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:46.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7931 cs=0 l=1 rev1=1 rx=0x7ff9940946b0 tx=0x7ff99400dfc0).stop 2021-03-09T10:40:46.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:46.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:46.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:46.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:46.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:46.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff99c0adc20 2021-03-09T10:40:46.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000aab0 con 0x7ff99c0613a0 2021-03-09T10:40:46.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015180 con 0x7ff98000ac10 2021-03-09T10:40:46.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:46.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:46.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:46.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:46.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.088+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:46.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.092+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:46.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.092+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:46.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.092+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:46.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.092+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:46.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.092+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800363d0 con 0x7ff98000ac10 2021-03-09T10:40:46.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.092+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980046b10 con 0x7ff98000ac10 2021-03-09T10:40:46.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.092+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=7936 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c02cf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:46.100 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:46.092+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26818 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:46.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08d030 con 0x7ff98000ac10 2021-03-09T10:40:46.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:40:46.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a9db0 con 0x7ff98000ac10 2021-03-09T10:40:46.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.100+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:46.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.100+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:46.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:46.100+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:51.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:51.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:51.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=7936 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c02cf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:51.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=7936 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c02cf90).stop 2021-03-09T10:40:51.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:51.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=CLOSED pgs=7936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:51.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:51.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:51.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:51.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000aab0 con 0x7ff99c0adc20 2021-03-09T10:40:51.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff99c0613a0 2021-03-09T10:40:51.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015320 con 0x7ff980051540 2021-03-09T10:40:51.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:51.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:51.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:51.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.092+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:51.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.096+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:51.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.096+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:51.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.096+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:51.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.096+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:51.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.096+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:51.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.096+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980046b10 con 0x7ff980051540 2021-03-09T10:40:51.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.096+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017490 con 0x7ff980051540 2021-03-09T10:40:51.105 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:51.096+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26853 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:51.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.096+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7947 cs=0 l=1 rev1=1 rx=0x7ff99406d5e0 tx=0x7ff99406d490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:51.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:40:51.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940700d0 con 0x7ff980051540 2021-03-09T10:40:51.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.104+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:40:51.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.104+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:51.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.104+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:51.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:51.104+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:40:56.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:56.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:56.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7947 cs=0 l=1 rev1=1 rx=0x7ff99406d5e0 tx=0x7ff99406d490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:56.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7947 cs=0 l=1 rev1=1 rx=0x7ff99406d5e0 tx=0x7ff99406d490).stop 2021-03-09T10:40:56.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:56.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:56.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:56.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:56.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:56.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010fb0 con 0x7ff99c0adc20 2021-03-09T10:40:56.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff99c0613a0 2021-03-09T10:40:56.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f220 con 0x7ff98000ac10 2021-03-09T10:40:56.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:40:56.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:56.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:56.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:40:56.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:56.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:56.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:56.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:56.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:56.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017630 con 0x7ff98000ac10 2021-03-09T10:40:56.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.100+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800363d0 con 0x7ff98000ac10 2021-03-09T10:40:56.108 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:40:56.100+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26892 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:40:56.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.104+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=7958 cs=0 l=1 rev1=1 rx=0x7ff98c08f000 tx=0x7ff98c005180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:56.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.104+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c009540 con 0x7ff98000ac10 2021-03-09T10:40:56.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.104+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c009540 con 0x7ff98000ac10 2021-03-09T10:40:56.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:40:56.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.108+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:56.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.108+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:56.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:40:56.108+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:01.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.100+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:01.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:01.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=7958 cs=0 l=1 rev1=1 rx=0x7ff98c08f000 tx=0x7ff98c005180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:01.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=7958 cs=0 l=1 rev1=1 rx=0x7ff98c08f000 tx=0x7ff98c005180).stop 2021-03-09T10:41:01.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:01.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=CLOSED pgs=7958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:01.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:01.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:01.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:01.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980010fb0 con 0x7ff99c0adc20 2021-03-09T10:41:01.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff99c0613a0 2021-03-09T10:41:01.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f220 con 0x7ff980051540 2021-03-09T10:41:01.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:01.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:01.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:01.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:01.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:01.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:01.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:01.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:01.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:01.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015180 con 0x7ff980051540 2021-03-09T10:41:01.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017630 con 0x7ff980051540 2021-03-09T10:41:01.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.104+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7968 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c022cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:01.114 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:01.108+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26926 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:01.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a9cd0 con 0x7ff980051540 2021-03-09T10:41:01.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c009540 con 0x7ff980051540 2021-03-09T10:41:01.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a9cd0 con 0x7ff980051540 2021-03-09T10:41:01.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.112+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:01.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.112+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:01.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:01.112+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:04.044 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:41:04.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:41:06.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:06.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:06.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7968 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c022cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:06.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7968 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c022cf0).stop 2021-03-09T10:41:06.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:06.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:06.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:06.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:06.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:06.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017630 con 0x7ff99c0adc20 2021-03-09T10:41:06.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800095f0 con 0x7ff99c0613a0 2021-03-09T10:41:06.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009730 con 0x7ff98000ac10 2021-03-09T10:41:06.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:06.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:06.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:06.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:06.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.108+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:06.160 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:06.112+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26965 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:06.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:06.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:06.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:06.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:06.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015310 con 0x7ff98000ac10 2021-03-09T10:41:06.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980046b10 con 0x7ff98000ac10 2021-03-09T10:41:06.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=7981 cs=0 l=1 rev1=1 rx=0x7ff980014050 tx=0x7ff994036960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:06.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.116+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff98000ac10 2021-03-09T10:41:06.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.116+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff98000ac10 2021-03-09T10:41:06.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.120+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99400cdb0 con 0x7ff98000ac10 2021-03-09T10:41:06.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.120+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:06.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.120+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:06.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:06.120+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:11.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:11.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:11.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=7981 cs=0 l=1 rev1=1 rx=0x7ff980014050 tx=0x7ff994036960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:11.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=7981 cs=0 l=1 rev1=1 rx=0x7ff980014050 tx=0x7ff994036960).stop 2021-03-09T10:41:11.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:11.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=CLOSED pgs=7981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:11.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:11.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:11.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:11.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a0b0 con 0x7ff99c0adc20 2021-03-09T10:41:11.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017630 con 0x7ff99c0613a0 2021-03-09T10:41:11.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800044f0 con 0x7ff980051540 2021-03-09T10:41:11.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:11.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:11.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:11.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:11.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.116+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:11.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.116+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:11.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.116+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:11.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.116+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:11.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.116+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:11.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.116+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980004660 con 0x7ff980051540 2021-03-09T10:41:11.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.116+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015180 con 0x7ff980051540 2021-03-09T10:41:11.125 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:11.116+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27000 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:11.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.116+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7991 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c00bf60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:11.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.116+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c09f170 con 0x7ff980051540 2021-03-09T10:41:11.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.120+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c022ed0 con 0x7ff980051540 2021-03-09T10:41:11.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.124+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:41:11.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.124+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:11.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.124+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:11.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:11.124+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:16.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:16.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:16.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7991 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c00bf60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:16.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=7991 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c00bf60).stop 2021-03-09T10:41:16.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:16.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=7991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:16.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:16.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:16.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:16.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013200 con 0x7ff99c0adc20 2021-03-09T10:41:16.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800044f0 con 0x7ff99c0613a0 2021-03-09T10:41:16.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017630 con 0x7ff98000ac10 2021-03-09T10:41:16.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:16.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:16.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:16.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:16.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:16.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:16.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:16.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:16.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:16.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002a0b0 con 0x7ff98000ac10 2021-03-09T10:41:16.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015310 con 0x7ff98000ac10 2021-03-09T10:41:16.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.119+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=8000 cs=0 l=1 rev1=1 rx=0x7ff980014bb0 tx=0x7ff994014fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:16.130 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:16.123+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27036 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:16.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.123+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff98000ac10 2021-03-09T10:41:16.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.123+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff98000ac10 2021-03-09T10:41:16.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.127+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:41:16.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.127+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:16.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.131+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:16.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:16.131+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:21.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:21.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:21.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=8000 cs=0 l=1 rev1=1 rx=0x7ff980014bb0 tx=0x7ff994014fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:21.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 secure :-1 s=READY pgs=8000 cs=0 l=1 rev1=1 rx=0x7ff980014bb0 tx=0x7ff994014fe0).stop 2021-03-09T10:41:21.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:21.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c90 unknown :-1 s=CLOSED pgs=8000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:21.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:21.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:21.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:21.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013200 con 0x7ff99c0adc20 2021-03-09T10:41:21.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011420 con 0x7ff99c0613a0 2021-03-09T10:41:21.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017630 con 0x7ff980051540 2021-03-09T10:41:21.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:21.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:21.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:21.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:21.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.123+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:21.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.127+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:21.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.127+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:21.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.127+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:21.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.127+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:21.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.127+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015180 con 0x7ff980051540 2021-03-09T10:41:21.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.127+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002a0b0 con 0x7ff980051540 2021-03-09T10:41:21.135 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:21.127+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27074 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:21.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.127+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=8012 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99406f0e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:21.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.131+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940aa080 con 0x7ff980051540 2021-03-09T10:41:21.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.135+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940aa080 con 0x7ff980051540 2021-03-09T10:41:21.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.135+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff980051540 2021-03-09T10:41:21.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.135+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:21.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.135+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:21.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:21.135+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:26.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:26.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:26.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=8012 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99406f0e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:26.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=8012 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99406f0e0).stop 2021-03-09T10:41:26.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:26.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=8012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:26.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:26.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:26.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:26.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092f30 con 0x7ff99c0adc20 2021-03-09T10:41:26.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a0b0 con 0x7ff99c0613a0 2021-03-09T10:41:26.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017380 con 0x7ff98000ac10 2021-03-09T10:41:26.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:26.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:26.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:26.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:26.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.127+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:26.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.131+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:26.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.131+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:26.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.131+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:26.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.131+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:26.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.131+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800174f0 con 0x7ff98000ac10 2021-03-09T10:41:26.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.131+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015180 con 0x7ff98000ac10 2021-03-09T10:41:26.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.131+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8023 cs=0 l=1 rev1=1 rx=0x7ff994034080 tx=0x7ff99400ae90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:26.140 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:26.131+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27110 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:26.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.135+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98000ac10 2021-03-09T10:41:26.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.135+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98000ac10 2021-03-09T10:41:26.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.139+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940343d0 con 0x7ff98000ac10 2021-03-09T10:41:26.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.139+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:26.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.139+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:26.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:26.139+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:31.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:31.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:31.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8023 cs=0 l=1 rev1=1 rx=0x7ff994034080 tx=0x7ff99400ae90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:31.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8023 cs=0 l=1 rev1=1 rx=0x7ff994034080 tx=0x7ff99400ae90).stop 2021-03-09T10:41:31.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:31.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=CLOSED pgs=8023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:31.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:31.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:31.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:31.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a0b0 con 0x7ff99c0adc20 2021-03-09T10:41:31.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017330 con 0x7ff99c0613a0 2021-03-09T10:41:31.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015160 con 0x7ff980051540 2021-03-09T10:41:31.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:31.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:31.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:31.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:31.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:31.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:31.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.135+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:31.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.139+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:31.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.139+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:31.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.139+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800152a0 con 0x7ff980051540 2021-03-09T10:41:31.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.139+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800174f0 con 0x7ff980051540 2021-03-09T10:41:31.146 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:31.139+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27148 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:31.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.139+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=8034 cs=0 l=1 rev1=1 rx=0x7ff99402abc0 tx=0x7ff99400bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:31.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.139+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408cc40 con 0x7ff980051540 2021-03-09T10:41:31.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.139+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408b630 con 0x7ff980051540 2021-03-09T10:41:31.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.143+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940343d0 con 0x7ff980051540 2021-03-09T10:41:31.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.143+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:31.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.147+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:31.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:31.147+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:34.090 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:41:34.093 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:41:36.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:36.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:36.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=8034 cs=0 l=1 rev1=1 rx=0x7ff99402abc0 tx=0x7ff99400bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:36.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 secure :-1 s=READY pgs=8034 cs=0 l=1 rev1=1 rx=0x7ff99402abc0 tx=0x7ff99400bfd0).stop 2021-03-09T10:41:36.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:36.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980025f90 unknown :-1 s=CLOSED pgs=8034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:36.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:36.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:36.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:36.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015160 con 0x7ff99c0adc20 2021-03-09T10:41:36.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000dae0 con 0x7ff99c0613a0 2021-03-09T10:41:36.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:36.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a0b0 con 0x7ff98000ac10 2021-03-09T10:41:36.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:36.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:36.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:36.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:36.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:36.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:36.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:36.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:36.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800172e0 con 0x7ff98000ac10 2021-03-09T10:41:36.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.139+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980004660 con 0x7ff98000ac10 2021-03-09T10:41:36.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.143+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8041 cs=0 l=1 rev1=1 rx=0x7ff9800131d0 tx=0x7ff98c00bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:36.152 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:36.143+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27184 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:36.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98000ac10 2021-03-09T10:41:36.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079fc0 con 0x7ff98000ac10 2021-03-09T10:41:36.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.151+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c085140 con 0x7ff98000ac10 2021-03-09T10:41:36.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.151+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:36.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.151+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:36.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:36.151+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:41.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:41.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:41.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8041 cs=0 l=1 rev1=1 rx=0x7ff9800131d0 tx=0x7ff98c00bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:41.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8041 cs=0 l=1 rev1=1 rx=0x7ff9800131d0 tx=0x7ff98c00bfd0).stop 2021-03-09T10:41:41.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:41.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=CLOSED pgs=8041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:41.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:41.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:41.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:41.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015160 con 0x7ff99c0adc20 2021-03-09T10:41:41.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000dae0 con 0x7ff99c0613a0 2021-03-09T10:41:41.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a0b0 con 0x7ff980051540 2021-03-09T10:41:41.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:41.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:41.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:41.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:41.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:41.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:41.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:41.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:41.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:41.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980004800 con 0x7ff980051540 2021-03-09T10:41:41.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800172e0 con 0x7ff980051540 2021-03-09T10:41:41.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8053 cs=0 l=1 rev1=1 rx=0x7ff980040910 tx=0x7ff98c017d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:41.156 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:41.147+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27222 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:41.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a9120 con 0x7ff980051540 2021-03-09T10:41:41.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.151+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c009540 con 0x7ff980051540 2021-03-09T10:41:41.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.151+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a9120 con 0x7ff980051540 2021-03-09T10:41:41.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.151+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:41.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.155+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:41.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:41.155+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:46.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:46.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:46.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8053 cs=0 l=1 rev1=1 rx=0x7ff980040910 tx=0x7ff98c017d50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:46.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8053 cs=0 l=1 rev1=1 rx=0x7ff980040910 tx=0x7ff98c017d50).stop 2021-03-09T10:41:46.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:46.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=CLOSED pgs=8053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:46.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:46.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:46.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:46.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015160 con 0x7ff99c0adc20 2021-03-09T10:41:46.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008bd20 con 0x7ff99c0613a0 2021-03-09T10:41:46.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000dae0 con 0x7ff98000ac10 2021-03-09T10:41:46.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:46.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:46.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:46.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.147+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:46.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.151+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:46.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.151+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:46.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.151+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:46.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.151+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:46.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.151+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:46.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.151+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002a0b0 con 0x7ff98000ac10 2021-03-09T10:41:46.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.151+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800152a0 con 0x7ff98000ac10 2021-03-09T10:41:46.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.151+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8063 cs=0 l=1 rev1=1 rx=0x7ff994068890 tx=0x7ff99400af00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:46.161 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:46.151+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27258 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:46.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.155+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400c150 con 0x7ff98000ac10 2021-03-09T10:41:46.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.155+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:41:46.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.155+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:41:46.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.155+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:46.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.159+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:46.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:46.159+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:51.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:51.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:51.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8063 cs=0 l=1 rev1=1 rx=0x7ff994068890 tx=0x7ff99400af00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:51.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8063 cs=0 l=1 rev1=1 rx=0x7ff994068890 tx=0x7ff99400af00).stop 2021-03-09T10:41:51.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:51.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=CLOSED pgs=8063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:51.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:51.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:51.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:51.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d760 con 0x7ff99c0adc20 2021-03-09T10:41:51.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020af0 con 0x7ff99c0613a0 2021-03-09T10:41:51.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015160 con 0x7ff980051540 2021-03-09T10:41:51.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:51.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:51.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:51.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:51.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:51.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:51.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:51.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:51.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:51.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017480 con 0x7ff980051540 2021-03-09T10:41:51.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.155+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002a0b0 con 0x7ff980051540 2021-03-09T10:41:51.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.159+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8074 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99402ab30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:51.166 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:51.159+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27296 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:51.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.163+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff980051540 2021-03-09T10:41:51.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.163+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081340 con 0x7ff980051540 2021-03-09T10:41:51.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.167+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:41:51.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.167+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:51.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.167+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:51.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:51.167+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:41:56.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:56.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:56.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8074 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99402ab30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:56.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8074 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99402ab30).stop 2021-03-09T10:41:56.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:56.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=CLOSED pgs=8074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:56.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:56.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:56.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:56.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:41:56.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001e6b0 con 0x7ff99c0adc20 2021-03-09T10:41:56.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:56.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d760 con 0x7ff99c0613a0 2021-03-09T10:41:56.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a0b0 con 0x7ff98000ac10 2021-03-09T10:41:56.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:41:56.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:56.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:56.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:56.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:56.169 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:56.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:56.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.159+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015160 con 0x7ff98000ac10 2021-03-09T10:41:56.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.163+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017480 con 0x7ff98000ac10 2021-03-09T10:41:56.171 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:41:56.163+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27332 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:41:56.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.163+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8088 cs=0 l=1 rev1=1 rx=0x7ff994016a90 tx=0x7ff99405dfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:56.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.167+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940604d0 con 0x7ff98000ac10 2021-03-09T10:41:56.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.167+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081340 con 0x7ff98000ac10 2021-03-09T10:41:56.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:41:56.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.171+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:56.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.171+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:56.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:41:56.171+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:01.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:01.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:01.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8088 cs=0 l=1 rev1=1 rx=0x7ff994016a90 tx=0x7ff99405dfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:01.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8088 cs=0 l=1 rev1=1 rx=0x7ff994016a90 tx=0x7ff99405dfd0).stop 2021-03-09T10:42:01.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:01.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=CLOSED pgs=8088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:01.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:01.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:01.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:01.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001e6b0 con 0x7ff99c0adc20 2021-03-09T10:42:01.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980028630 con 0x7ff99c0613a0 2021-03-09T10:42:01.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015490 con 0x7ff980051540 2021-03-09T10:42:01.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:01.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:01.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.163+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:01.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.167+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:01.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.167+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:01.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.167+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:01.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.167+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:01.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.167+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:01.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.167+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:01.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.167+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002a0b0 con 0x7ff980051540 2021-03-09T10:42:01.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.167+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015160 con 0x7ff980051540 2021-03-09T10:42:01.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.167+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8095 cs=0 l=1 rev1=1 rx=0x7ff99405e040 tx=0x7ff994068220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:01.177 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:01.167+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27370 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:01.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:42:01.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940700e0 con 0x7ff980051540 2021-03-09T10:42:01.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:42:01.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.175+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:01.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.175+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:01.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:01.175+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:04.141 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:42:04.144 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:42:06.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:06.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:06.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8095 cs=0 l=1 rev1=1 rx=0x7ff99405e040 tx=0x7ff994068220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:06.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8095 cs=0 l=1 rev1=1 rx=0x7ff99405e040 tx=0x7ff994068220).stop 2021-03-09T10:42:06.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:06.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=CLOSED pgs=8095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:06.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:06.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:06.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:06.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001e6b0 con 0x7ff99c0adc20 2021-03-09T10:42:06.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022b50 con 0x7ff99c0613a0 2021-03-09T10:42:06.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d760 con 0x7ff98000ac10 2021-03-09T10:42:06.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:06.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:06.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:06.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:06.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:06.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:06.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:06.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:06.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:06.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015160 con 0x7ff98000ac10 2021-03-09T10:42:06.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.171+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980004800 con 0x7ff98000ac10 2021-03-09T10:42:06.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.175+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8105 cs=0 l=1 rev1=1 rx=0x7ff98c04d000 tx=0x7ff98c09f170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:06.182 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:06.175+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27406 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:06.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:42:06.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c099ed0 con 0x7ff98000ac10 2021-03-09T10:42:06.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.179+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:42:06.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.179+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:06.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.183+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:06.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:06.183+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:11.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:11.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:11.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8105 cs=0 l=1 rev1=1 rx=0x7ff98c04d000 tx=0x7ff98c09f170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:11.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 secure :-1 s=READY pgs=8105 cs=0 l=1 rev1=1 rx=0x7ff98c04d000 tx=0x7ff98c09f170).stop 2021-03-09T10:42:11.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:11.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014c70 unknown :-1 s=CLOSED pgs=8105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:11.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:11.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:11.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:11.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022b50 con 0x7ff99c0adc20 2021-03-09T10:42:11.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015490 con 0x7ff99c0613a0 2021-03-09T10:42:11.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c3f0 con 0x7ff980051540 2021-03-09T10:42:11.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:11.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:11.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:11.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:11.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.175+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:11.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.179+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:11.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.179+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:11.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.179+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:11.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.179+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:11.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.179+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020af0 con 0x7ff980051540 2021-03-09T10:42:11.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.179+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015160 con 0x7ff980051540 2021-03-09T10:42:11.192 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:11.179+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27444 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:11.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.179+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8120 cs=0 l=1 rev1=1 rx=0x7ff98c02a040 tx=0x7ff98c079e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:11.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980051540 2021-03-09T10:42:11.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079b60 con 0x7ff980051540 2021-03-09T10:42:11.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:42:11.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.187+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:11.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.187+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:11.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:11.187+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:16.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:16.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:16.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8120 cs=0 l=1 rev1=1 rx=0x7ff98c02a040 tx=0x7ff98c079e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:16.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8120 cs=0 l=1 rev1=1 rx=0x7ff98c02a040 tx=0x7ff98c079e60).stop 2021-03-09T10:42:16.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:16.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=CLOSED pgs=8120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:16.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:16.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:16.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:16.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001e6b0 con 0x7ff99c0adc20 2021-03-09T10:42:16.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022b50 con 0x7ff99c0613a0 2021-03-09T10:42:16.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015490 con 0x7ff98000ac10 2021-03-09T10:42:16.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:16.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:16.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:16.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:16.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:16.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:16.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:16.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:16.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:16.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020af0 con 0x7ff98000ac10 2021-03-09T10:42:16.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015160 con 0x7ff98000ac10 2021-03-09T10:42:16.193 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:16.183+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27480 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:16.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 secure :-1 s=READY pgs=8127 cs=0 l=1 rev1=1 rx=0x7ff98c017e90 tx=0x7ff98c086160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:16.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff98000ac10 2021-03-09T10:42:16.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c014650 con 0x7ff98000ac10 2021-03-09T10:42:16.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff98000ac10 2021-03-09T10:42:16.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.191+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:16.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.191+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:16.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:16.191+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:21.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:21.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.183+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:21.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 secure :-1 s=READY pgs=8127 cs=0 l=1 rev1=1 rx=0x7ff98c017e90 tx=0x7ff98c086160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:21.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.183+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 secure :-1 s=READY pgs=8127 cs=0 l=1 rev1=1 rx=0x7ff98c017e90 tx=0x7ff98c086160).stop 2021-03-09T10:42:21.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.183+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800968b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:21.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 unknown :-1 s=CLOSED pgs=8127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:21.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:21.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.183+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:21.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:21.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c3f0 con 0x7ff99c0adc20 2021-03-09T10:42:21.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022b50 con 0x7ff99c0613a0 2021-03-09T10:42:21.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015490 con 0x7ff980051540 2021-03-09T10:42:21.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:21.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:21.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:21.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:21.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:21.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:21.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:21.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:21.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:21.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800147e0 con 0x7ff980051540 2021-03-09T10:42:21.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.187+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015160 con 0x7ff980051540 2021-03-09T10:42:21.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.191+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8137 cs=0 l=1 rev1=1 rx=0x7ff98c01d040 tx=0x7ff98c020200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:21.198 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:21.191+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27518 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:21.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff980051540 2021-03-09T10:42:21.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04a000 con 0x7ff980051540 2021-03-09T10:42:21.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079b60 con 0x7ff980051540 2021-03-09T10:42:21.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.199+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:21.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.199+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:21.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:21.199+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:26.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:26.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:26.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8137 cs=0 l=1 rev1=1 rx=0x7ff98c01d040 tx=0x7ff98c020200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:26.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8137 cs=0 l=1 rev1=1 rx=0x7ff98c01d040 tx=0x7ff98c020200).stop 2021-03-09T10:42:26.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:26.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=CLOSED pgs=8137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:26.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800290c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:26.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:26.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:26.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001e6b0 con 0x7ff99c0adc20 2021-03-09T10:42:26.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c3f0 con 0x7ff99c0613a0 2021-03-09T10:42:26.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800078e0 con 0x7ff98000ac10 2021-03-09T10:42:26.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:26.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:26.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:26.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:26.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.191+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800290c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:26.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:26.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:26.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:26.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:26.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015160 con 0x7ff98000ac10 2021-03-09T10:42:26.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.195+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002a0b0 con 0x7ff98000ac10 2021-03-09T10:42:26.203 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:26.195+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27554 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:26.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.195+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800290c0 secure :-1 s=READY pgs=8153 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940944f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:26.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.195+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:42:26.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.199+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:42:26.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.203+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:42:26.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.203+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:26.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.203+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:26.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:26.203+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:31.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800290c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:31.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800290c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:31.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800290c0 secure :-1 s=READY pgs=8153 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940944f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:31.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800290c0 secure :-1 s=READY pgs=8153 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff9940944f0).stop 2021-03-09T10:42:31.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800290c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:31.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800290c0 unknown :-1 s=CLOSED pgs=8153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:31.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:31.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:31.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:31.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001e6b0 con 0x7ff99c0adc20 2021-03-09T10:42:31.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015490 con 0x7ff99c0613a0 2021-03-09T10:42:31.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800078e0 con 0x7ff980051540 2021-03-09T10:42:31.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:31.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:31.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:31.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:31.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.194+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:31.206 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:31.198+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27592 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:31.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.198+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:31.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.198+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:31.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.198+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:31.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.198+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:31.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.198+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020ac0 con 0x7ff980051540 2021-03-09T10:42:31.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.198+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015160 con 0x7ff980051540 2021-03-09T10:42:31.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.202+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8164 cs=0 l=1 rev1=1 rx=0x7ff9940982e0 tx=0x7ff99408d1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:31.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff980051540 2021-03-09T10:42:31.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994083300 con 0x7ff980051540 2021-03-09T10:42:31.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff980051540 2021-03-09T10:42:31.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.206+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:31.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.206+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:31.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:31.206+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:34.197 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:42:34.205 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:42:36.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:36.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:36.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8164 cs=0 l=1 rev1=1 rx=0x7ff9940982e0 tx=0x7ff99408d1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:36.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8164 cs=0 l=1 rev1=1 rx=0x7ff9940982e0 tx=0x7ff99408d1f0).stop 2021-03-09T10:42:36.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:36.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=CLOSED pgs=8164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:36.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:36.208 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:36.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:36.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:36.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98001e6b0 con 0x7ff99c0adc20 2021-03-09T10:42:36.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:36.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000dd30 con 0x7ff99c0613a0 2021-03-09T10:42:36.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.198+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015490 con 0x7ff98000ac10 2021-03-09T10:42:36.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.202+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:36.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.202+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:36.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.202+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:36.212 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:36.202+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27628 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:36.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:36.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:36.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:36.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:36.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020ac0 con 0x7ff98000ac10 2021-03-09T10:42:36.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015160 con 0x7ff98000ac10 2021-03-09T10:42:36.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028ed0 secure :-1 s=READY pgs=8175 cs=0 l=1 rev1=1 rx=0x7ff99405d250 tx=0x7ff99405d330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:36.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff98000ac10 2021-03-09T10:42:36.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994030740 con 0x7ff98000ac10 2021-03-09T10:42:36.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff98000ac10 2021-03-09T10:42:36.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.210+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:36.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.210+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:36.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:36.210+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:41.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980028ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:41.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980028ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:41.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028ed0 secure :-1 s=READY pgs=8175 cs=0 l=1 rev1=1 rx=0x7ff99405d250 tx=0x7ff99405d330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:41.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028ed0 secure :-1 s=READY pgs=8175 cs=0 l=1 rev1=1 rx=0x7ff99405d250 tx=0x7ff99405d330).stop 2021-03-09T10:42:41.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980028ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:41.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028ed0 unknown :-1 s=CLOSED pgs=8175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:41.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:41.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:41.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:41.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000dd30 con 0x7ff99c0adc20 2021-03-09T10:42:41.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005480 con 0x7ff99c0613a0 2021-03-09T10:42:41.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020ac0 con 0x7ff980051540 2021-03-09T10:42:41.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:41.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:41.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:41.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:41.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:41.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:41.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:41.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:41.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:41.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.206+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008ffe0 con 0x7ff980051540 2021-03-09T10:42:41.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.210+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015160 con 0x7ff980051540 2021-03-09T10:42:41.217 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:41.206+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27666 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:41.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.210+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8184 cs=0 l=1 rev1=1 rx=0x7ff98004d000 tx=0x7ff9940983c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:41.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095120 con 0x7ff980051540 2021-03-09T10:42:41.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994029ce0 con 0x7ff980051540 2021-03-09T10:42:41.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff980051540 2021-03-09T10:42:41.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.218+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:41.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.218+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:41.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:41.218+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:46.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:46.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:46.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8184 cs=0 l=1 rev1=1 rx=0x7ff98004d000 tx=0x7ff9940983c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:46.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8184 cs=0 l=1 rev1=1 rx=0x7ff98004d000 tx=0x7ff9940983c0).stop 2021-03-09T10:42:46.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:46.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=CLOSED pgs=8184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:46.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:46.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:46.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:46.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005480 con 0x7ff99c0adc20 2021-03-09T10:42:46.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000dce0 con 0x7ff99c0613a0 2021-03-09T10:42:46.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029340 con 0x7ff98000ac10 2021-03-09T10:42:46.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:46.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:46.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:46.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:46.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.210+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:46.223 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:46.214+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27700 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:46.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:46.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:46.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:46.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:46.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015100 con 0x7ff98000ac10 2021-03-09T10:42:46.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800147e0 con 0x7ff98000ac10 2021-03-09T10:42:46.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.218+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028e50 secure :-1 s=READY pgs=8196 cs=0 l=1 rev1=1 rx=0x7ff98c0052d0 tx=0x7ff98c00b4d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:46.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00ba80 con 0x7ff98000ac10 2021-03-09T10:42:46.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00ba80 con 0x7ff98000ac10 2021-03-09T10:42:46.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:42:46.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.222+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:46.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.226+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:46.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:46.226+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:51.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980028e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:51.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980028e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:51.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028e50 secure :-1 s=READY pgs=8196 cs=0 l=1 rev1=1 rx=0x7ff98c0052d0 tx=0x7ff98c00b4d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:51.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028e50 secure :-1 s=READY pgs=8196 cs=0 l=1 rev1=1 rx=0x7ff98c0052d0 tx=0x7ff98c00b4d0).stop 2021-03-09T10:42:51.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980028e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:51.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028e50 unknown :-1 s=CLOSED pgs=8196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:51.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:51.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:51.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:51.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029340 con 0x7ff99c0adc20 2021-03-09T10:42:51.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020ac0 con 0x7ff99c0613a0 2021-03-09T10:42:51.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048400 con 0x7ff980051540 2021-03-09T10:42:51.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:51.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:51.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:51.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:51.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:51.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:51.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:51.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:51.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:51.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980048540 con 0x7ff980051540 2021-03-09T10:42:51.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.214+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015100 con 0x7ff980051540 2021-03-09T10:42:51.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.218+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8201 cs=0 l=1 rev1=1 rx=0x7ff98c04fa40 tx=0x7ff98c022ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:51.228 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:51.218+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27740 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:51.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.218+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c03b890 con 0x7ff980051540 2021-03-09T10:42:51.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03b890 con 0x7ff980051540 2021-03-09T10:42:51.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980051540 2021-03-09T10:42:51.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.226+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:51.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.226+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:51.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:51.226+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:42:56.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:56.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:56.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8201 cs=0 l=1 rev1=1 rx=0x7ff98c04fa40 tx=0x7ff98c022ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:56.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 secure :-1 s=READY pgs=8201 cs=0 l=1 rev1=1 rx=0x7ff98c04fa40 tx=0x7ff98c022ed0).stop 2021-03-09T10:42:56.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980023f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:56.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980023f80 unknown :-1 s=CLOSED pgs=8201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:56.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:56.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:56.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:56.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048400 con 0x7ff99c0adc20 2021-03-09T10:42:56.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005840 con 0x7ff99c0613a0 2021-03-09T10:42:56.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800152a0 con 0x7ff98000ac10 2021-03-09T10:42:56.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:42:56.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:56.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:42:56.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:56.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:56.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:56.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:56.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:56.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:56.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980029150 con 0x7ff98000ac10 2021-03-09T10:42:56.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980048540 con 0x7ff98000ac10 2021-03-09T10:42:56.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028c60 secure :-1 s=READY pgs=8213 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c005f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:56.233 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:42:56.222+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27776 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:42:56.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c005cd0 con 0x7ff98000ac10 2021-03-09T10:42:56.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.226+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08f360 con 0x7ff98000ac10 2021-03-09T10:42:56.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:42:56.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.230+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:56.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.230+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:56.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:42:56.230+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:01.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980028c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:01.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980028c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:01.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028c60 secure :-1 s=READY pgs=8213 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c005f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:01.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028c60 secure :-1 s=READY pgs=8213 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c005f20).stop 2021-03-09T10:43:01.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980028c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:01.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980028c60 unknown :-1 s=CLOSED pgs=8213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:01.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:01.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:01.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:01.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005840 con 0x7ff99c0adc20 2021-03-09T10:43:01.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800152c0 con 0x7ff99c0613a0 2021-03-09T10:43:01.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020ac0 con 0x7ff980051540 2021-03-09T10:43:01.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.222+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:01.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:01.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:01.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:01.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:01.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:01.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:01.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:01.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:01.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800485b0 con 0x7ff980051540 2021-03-09T10:43:01.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ffe0 con 0x7ff980051540 2021-03-09T10:43:01.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.226+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800480c0 secure :-1 s=READY pgs=8223 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99400bf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:01.238 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:01.226+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27814 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:01.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:43:01.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940700d0 con 0x7ff980051540 2021-03-09T10:43:01.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.234+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:43:01.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.234+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:01.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.234+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:01.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:01.234+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:04.262 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:43:04.265 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:43:06.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:43:06.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:06.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800480c0 secure :-1 s=READY pgs=8223 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99400bf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:06.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800480c0 secure :-1 s=READY pgs=8223 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99400bf40).stop 2021-03-09T10:43:06.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800480c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:06.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800480c0 unknown :-1 s=CLOSED pgs=8223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:06.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:06.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:06.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:06.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020c60 con 0x7ff99c0adc20 2021-03-09T10:43:06.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980089170 con 0x7ff99c0613a0 2021-03-09T10:43:06.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005840 con 0x7ff98000ac10 2021-03-09T10:43:06.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:06.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:06.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:06.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:06.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:06.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:06.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:06.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:06.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:06.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:43:06.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800485b0 con 0x7ff98000ac10 2021-03-09T10:43:06.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.230+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8233 cs=0 l=1 rev1=1 rx=0x7ff9940142e0 tx=0x7ff99408d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:06.244 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:06.234+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27849 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:06.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.234+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009a80 con 0x7ff98000ac10 2021-03-09T10:43:06.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.234+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994070080 con 0x7ff98000ac10 2021-03-09T10:43:06.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.242+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff98000ac10 2021-03-09T10:43:06.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.242+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:06.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.242+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:06.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:06.242+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:11.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:11.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:11.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8233 cs=0 l=1 rev1=1 rx=0x7ff9940142e0 tx=0x7ff99408d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:11.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8233 cs=0 l=1 rev1=1 rx=0x7ff9940142e0 tx=0x7ff99408d180).stop 2021-03-09T10:43:11.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:11.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=CLOSED pgs=8233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:11.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:11.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:11.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:11.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029b20 con 0x7ff99c0adc20 2021-03-09T10:43:11.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980089170 con 0x7ff99c0613a0 2021-03-09T10:43:11.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048560 con 0x7ff980051540 2021-03-09T10:43:11.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:11.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:11.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:11.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:11.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:11.245 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:11.238+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27887 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:11.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:11.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:11.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:11.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:11.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800486a0 con 0x7ff980051540 2021-03-09T10:43:11.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.238+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff980051540 2021-03-09T10:43:11.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.242+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048070 secure :-1 s=READY pgs=8250 cs=0 l=1 rev1=1 rx=0x7ff99400dfb0 tx=0x7ff99400ac30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:11.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.242+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009a80 con 0x7ff980051540 2021-03-09T10:43:11.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.246+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994012df0 con 0x7ff980051540 2021-03-09T10:43:11.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.246+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070030 con 0x7ff980051540 2021-03-09T10:43:11.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.246+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:11.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.246+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:11.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:11.246+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:16.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:43:16.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:16.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048070 secure :-1 s=READY pgs=8250 cs=0 l=1 rev1=1 rx=0x7ff99400dfb0 tx=0x7ff99400ac30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:16.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048070 secure :-1 s=READY pgs=8250 cs=0 l=1 rev1=1 rx=0x7ff99400dfb0 tx=0x7ff99400ac30).stop 2021-03-09T10:43:16.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:16.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048070 unknown :-1 s=CLOSED pgs=8250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:16.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:16.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:16.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:16.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048560 con 0x7ff99c0adc20 2021-03-09T10:43:16.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020ca0 con 0x7ff99c0613a0 2021-03-09T10:43:16.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047040 con 0x7ff98000ac10 2021-03-09T10:43:16.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:16.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:16.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:16.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:16.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.242+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:16.251 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:16.242+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27924 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:16.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.246+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:16.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.246+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:16.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.246+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:16.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.246+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:16.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.246+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:43:16.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.246+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980029150 con 0x7ff98000ac10 2021-03-09T10:43:16.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.246+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8260 cs=0 l=1 rev1=1 rx=0x7ff98c04c000 tx=0x7ff98c079700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:16.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.250+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a0130 con 0x7ff98000ac10 2021-03-09T10:43:16.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.250+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff98000ac10 2021-03-09T10:43:16.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.254+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:43:16.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.254+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:16.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.254+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:16.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:16.254+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:21.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:21.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:21.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8260 cs=0 l=1 rev1=1 rx=0x7ff98c04c000 tx=0x7ff98c079700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:21.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8260 cs=0 l=1 rev1=1 rx=0x7ff98c04c000 tx=0x7ff98c079700).stop 2021-03-09T10:43:21.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:21.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=CLOSED pgs=8260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:21.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:21.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:21.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:21.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff99c0adc20 2021-03-09T10:43:21.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020aa0 con 0x7ff99c0613a0 2021-03-09T10:43:21.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020be0 con 0x7ff980051540 2021-03-09T10:43:21.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:21.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:21.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:21.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:21.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.246+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:21.256 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:21.250+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27961 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:21.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.250+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:21.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.250+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:21.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.250+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:21.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.250+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:21.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.250+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096f30 con 0x7ff980051540 2021-03-09T10:43:21.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.250+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800486a0 con 0x7ff980051540 2021-03-09T10:43:21.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.250+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096a40 secure :-1 s=READY pgs=8271 cs=0 l=1 rev1=1 rx=0x7ff9940b16d0 tx=0x7ff99405d090).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:21.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.254+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940aa080 con 0x7ff980051540 2021-03-09T10:43:21.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940b1530 con 0x7ff980051540 2021-03-09T10:43:21.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:43:21.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.258+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:21.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.258+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:21.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:21.258+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:26.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980096a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:26.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980096a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:26.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096a40 secure :-1 s=READY pgs=8271 cs=0 l=1 rev1=1 rx=0x7ff9940b16d0 tx=0x7ff99405d090).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:26.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096a40 secure :-1 s=READY pgs=8271 cs=0 l=1 rev1=1 rx=0x7ff9940b16d0 tx=0x7ff99405d090).stop 2021-03-09T10:43:26.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980096a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:26.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096a40 unknown :-1 s=CLOSED pgs=8271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:26.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:26.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:26.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:26.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011cb0 con 0x7ff99c0adc20 2021-03-09T10:43:26.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017fe0 con 0x7ff99c0613a0 2021-03-09T10:43:26.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800473f0 con 0x7ff98000ac10 2021-03-09T10:43:26.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:26.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:26.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:26.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:26.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.250+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:26.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.254+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:26.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.254+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:26.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.254+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:26.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.254+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:26.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.254+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800970c0 con 0x7ff98000ac10 2021-03-09T10:43:26.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.254+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096f30 con 0x7ff98000ac10 2021-03-09T10:43:26.263 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:26.254+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27997 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:26.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.254+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8280 cs=0 l=1 rev1=1 rx=0x7ff99408d2e0 tx=0x7ff994014210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:26.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.254+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99404a020 con 0x7ff98000ac10 2021-03-09T10:43:26.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99404a020 con 0x7ff98000ac10 2021-03-09T10:43:26.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.262+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070030 con 0x7ff98000ac10 2021-03-09T10:43:26.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.262+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:26.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.262+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:26.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:26.262+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:31.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:31.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:31.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8280 cs=0 l=1 rev1=1 rx=0x7ff99408d2e0 tx=0x7ff994014210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:31.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8280 cs=0 l=1 rev1=1 rx=0x7ff99408d2e0 tx=0x7ff994014210).stop 2021-03-09T10:43:31.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:31.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=CLOSED pgs=8280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:31.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:31.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:31.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:31.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017fe0 con 0x7ff99c0adc20 2021-03-09T10:43:31.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980096f80 con 0x7ff99c0613a0 2021-03-09T10:43:31.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a120 con 0x7ff980051540 2021-03-09T10:43:31.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:31.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:31.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:31.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:31.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:31.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:31.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:31.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:31.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:31.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092aa0 con 0x7ff980051540 2021-03-09T10:43:31.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800970c0 con 0x7ff980051540 2021-03-09T10:43:31.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097250 secure :-1 s=READY pgs=8287 cs=0 l=1 rev1=1 rx=0x7ff994000fa0 tx=0x7ff99406d1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:31.268 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:31.258+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28035 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:31.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994068ed0 con 0x7ff980051540 2021-03-09T10:43:31.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980051540 2021-03-09T10:43:31.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.262+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070030 con 0x7ff980051540 2021-03-09T10:43:31.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.262+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:31.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.266+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:31.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:31.266+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:34.301 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:43:34.311 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:43:36.266 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980097250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:36.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980097250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:36.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097250 secure :-1 s=READY pgs=8287 cs=0 l=1 rev1=1 rx=0x7ff994000fa0 tx=0x7ff99406d1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:36.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097250 secure :-1 s=READY pgs=8287 cs=0 l=1 rev1=1 rx=0x7ff994000fa0 tx=0x7ff99406d1f0).stop 2021-03-09T10:43:36.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980097250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:36.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980097250 unknown :-1 s=CLOSED pgs=8287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:36.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:36.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:36.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:36.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011cb0 con 0x7ff99c0adc20 2021-03-09T10:43:36.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980097100 con 0x7ff99c0613a0 2021-03-09T10:43:36.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800156e0 con 0x7ff98000ac10 2021-03-09T10:43:36.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:36.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:36.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:36.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.258+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:36.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.262+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:36.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.262+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:36.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.262+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:36.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.262+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:36.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.262+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:36.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.262+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800480b0 con 0x7ff98000ac10 2021-03-09T10:43:36.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.262+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:43:36.273 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:36.262+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28072 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:36.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.266+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8301 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0067d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:36.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.270+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98000ac10 2021-03-09T10:43:36.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.270+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079e60 con 0x7ff98000ac10 2021-03-09T10:43:36.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.274+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:43:36.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.274+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:36.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.274+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:36.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:36.274+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:41.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:41.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:41.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8301 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0067d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:41.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8301 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0067d0).stop 2021-03-09T10:43:41.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:41.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=CLOSED pgs=8301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:41.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:41.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:41.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:41.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023710 con 0x7ff99c0adc20 2021-03-09T10:43:41.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a120 con 0x7ff99c0613a0 2021-03-09T10:43:41.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000e120 con 0x7ff980051540 2021-03-09T10:43:41.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:41.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:41.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:41.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:41.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.266+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:41.283 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:41.270+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28109 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:41.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.270+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:41.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.270+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:41.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.270+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:41.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.270+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:41.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.270+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:43:41.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.270+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800480b0 con 0x7ff980051540 2021-03-09T10:43:41.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.270+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 secure :-1 s=READY pgs=8316 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c00b450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:41.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.274+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a9120 con 0x7ff980051540 2021-03-09T10:43:41.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.278+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a9120 con 0x7ff980051540 2021-03-09T10:43:41.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.278+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:43:41.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.278+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:41.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.278+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:41.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:41.278+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:46.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:43:46.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:46.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 secure :-1 s=READY pgs=8316 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c00b450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:46.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 secure :-1 s=READY pgs=8316 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c00b450).stop 2021-03-09T10:43:46.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:46.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 unknown :-1 s=CLOSED pgs=8316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:46.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:46.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:46.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:46.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:46.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:46.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000e120 con 0x7ff99c0adc20 2021-03-09T10:43:46.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023710 con 0x7ff99c0613a0 2021-03-09T10:43:46.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003a10 con 0x7ff98000ac10 2021-03-09T10:43:46.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:46.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:46.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:46.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:46.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:46.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:46.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:46.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:43:46.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092aa0 con 0x7ff98000ac10 2021-03-09T10:43:46.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8319 cs=0 l=1 rev1=1 rx=0x7ff994081e00 tx=0x7ff9940944d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:46.283 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:46.273+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28145 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:46.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.273+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:43:46.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.281+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:43:46.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.281+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:43:46.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.281+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:46.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.281+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:46.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:46.281+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:51.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:51.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:51.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8319 cs=0 l=1 rev1=1 rx=0x7ff994081e00 tx=0x7ff9940944d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:51.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8319 cs=0 l=1 rev1=1 rx=0x7ff994081e00 tx=0x7ff9940944d0).stop 2021-03-09T10:43:51.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:51.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=CLOSED pgs=8319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:51.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:51.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:51.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:51.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:51.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056c00 con 0x7ff99c0adc20 2021-03-09T10:43:51.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:51.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023710 con 0x7ff99c0613a0 2021-03-09T10:43:51.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002a120 con 0x7ff980051540 2021-03-09T10:43:51.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:51.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:51.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:51.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:51.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:51.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:51.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:51.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98001bf80 con 0x7ff980051540 2021-03-09T10:43:51.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092aa0 con 0x7ff980051540 2021-03-09T10:43:51.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.277+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 secure :-1 s=READY pgs=8329 cs=0 l=1 rev1=1 rx=0x7ff980027a80 tx=0x7ff99409d280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:51.288 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:51.277+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28183 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:51.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.281+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408c020 con 0x7ff980051540 2021-03-09T10:43:51.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.281+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980051540 2021-03-09T10:43:51.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.285+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:43:51.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.285+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:51.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.285+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:51.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:51.285+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:43:56.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:43:56.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:56.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 secure :-1 s=READY pgs=8329 cs=0 l=1 rev1=1 rx=0x7ff980027a80 tx=0x7ff99409d280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:56.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 secure :-1 s=READY pgs=8329 cs=0 l=1 rev1=1 rx=0x7ff980027a80 tx=0x7ff99409d280).stop 2021-03-09T10:43:56.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:56.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 unknown :-1 s=CLOSED pgs=8329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:56.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:56.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:56.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:56.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047190 con 0x7ff99c0adc20 2021-03-09T10:43:56.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023710 con 0x7ff99c0613a0 2021-03-09T10:43:56.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800555b0 con 0x7ff98000ac10 2021-03-09T10:43:56.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:43:56.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:56.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:43:56.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:56.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:56.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:56.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:56.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:56.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:56.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980003830 con 0x7ff98000ac10 2021-03-09T10:43:56.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.281+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98001bf80 con 0x7ff98000ac10 2021-03-09T10:43:56.294 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:43:56.281+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28220 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:43:56.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.285+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8345 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99408d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:56.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.285+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940ae1f0 con 0x7ff98000ac10 2021-03-09T10:43:56.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.289+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98000ac10 2021-03-09T10:43:56.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.293+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:43:56.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.293+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:56.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.293+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:56.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:43:56.293+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:01.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:01.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:01.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8345 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99408d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:01.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8345 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99408d180).stop 2021-03-09T10:44:01.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:01.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=CLOSED pgs=8345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:01.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:01.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:01.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:01.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023710 con 0x7ff99c0adc20 2021-03-09T10:44:01.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004730 con 0x7ff99c0613a0 2021-03-09T10:44:01.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800555b0 con 0x7ff980051540 2021-03-09T10:44:01.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:01.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:01.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:01.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:01.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:01.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:01.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:01.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:01.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:01.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:44:01.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.285+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800480b0 con 0x7ff980051540 2021-03-09T10:44:01.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.289+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 secure :-1 s=READY pgs=8352 cs=0 l=1 rev1=1 rx=0x7ff98c04c000 tx=0x7ff98c079770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:01.299 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:01.289+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28257 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:01.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.293+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:44:01.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.293+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:44:01.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.297+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:44:01.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.297+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:01.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.297+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:01.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:01.297+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:04.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:44:04.359 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:44:06.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:06.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:06.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 secure :-1 s=READY pgs=8352 cs=0 l=1 rev1=1 rx=0x7ff98c04c000 tx=0x7ff98c079770).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:06.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 secure :-1 s=READY pgs=8352 cs=0 l=1 rev1=1 rx=0x7ff98c04c000 tx=0x7ff98c079770).stop 2021-03-09T10:44:06.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980048240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:06.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980048240 unknown :-1 s=CLOSED pgs=8352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:06.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:06.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:06.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:06.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004730 con 0x7ff99c0adc20 2021-03-09T10:44:06.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023710 con 0x7ff99c0613a0 2021-03-09T10:44:06.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800555b0 con 0x7ff98000ac10 2021-03-09T10:44:06.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:06.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:06.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:06.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:06.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:06.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:06.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:06.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:06.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:06.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:44:06.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980003830 con 0x7ff98000ac10 2021-03-09T10:44:06.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.293+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8361 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940845b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:06.304 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:06.293+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28294 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:06.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.297+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98000ac10 2021-03-09T10:44:06.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.297+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99402a440 con 0x7ff98000ac10 2021-03-09T10:44:06.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.301+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:44:06.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.301+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:06.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.301+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:06.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:06.301+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:11.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.297+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:11.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.297+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:11.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.297+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8361 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940845b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:11.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.297+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8361 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940845b0).stop 2021-03-09T10:44:11.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:11.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=CLOSED pgs=8361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:11.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:11.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:11.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:11.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004730 con 0x7ff99c0adc20 2021-03-09T10:44:11.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023a30 con 0x7ff99c0613a0 2021-03-09T10:44:11.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011000 con 0x7ff980051540 2021-03-09T10:44:11.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:11.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:11.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:11.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:11.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.301+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:11.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.305+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:11.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.305+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:11.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.305+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:11.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.305+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:11.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.305+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:44:11.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.305+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980003830 con 0x7ff980051540 2021-03-09T10:44:11.310 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:11.305+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28331 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:11.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.305+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 secure :-1 s=READY pgs=8380 cs=0 l=1 rev1=1 rx=0x7ff994001530 tx=0x7ff99409bdc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:11.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.309+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400cd90 con 0x7ff980051540 2021-03-09T10:44:11.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.313+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:44:11.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.313+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99400cd90 con 0x7ff980051540 2021-03-09T10:44:11.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.313+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:11.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.313+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:11.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:11.313+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:16.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:16.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:16.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 secure :-1 s=READY pgs=8380 cs=0 l=1 rev1=1 rx=0x7ff994001530 tx=0x7ff99409bdc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:16.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 secure :-1 s=READY pgs=8380 cs=0 l=1 rev1=1 rx=0x7ff994001530 tx=0x7ff99409bdc0).stop 2021-03-09T10:44:16.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:16.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 unknown :-1 s=CLOSED pgs=8380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:16.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:16.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:16.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:16.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023710 con 0x7ff99c0adc20 2021-03-09T10:44:16.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004730 con 0x7ff99c0613a0 2021-03-09T10:44:16.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011000 con 0x7ff98000ac10 2021-03-09T10:44:16.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:16.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:16.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:16.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.305+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:16.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.309+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:16.314 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:16.309+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28368 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:16.314 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.309+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:16.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.309+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:16.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.309+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:16.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.309+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:16.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.309+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:44:16.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.309+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800480b0 con 0x7ff98000ac10 2021-03-09T10:44:16.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.313+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8389 cs=0 l=1 rev1=1 rx=0x7ff980045400 tx=0x7ff98c003ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:16.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.313+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff98000ac10 2021-03-09T10:44:16.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.321+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c009540 con 0x7ff98000ac10 2021-03-09T10:44:16.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.321+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff98000ac10 2021-03-09T10:44:16.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.321+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:16.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.321+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:16.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:16.321+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:21.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:21.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:21.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8389 cs=0 l=1 rev1=1 rx=0x7ff980045400 tx=0x7ff98c003ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:21.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 secure :-1 s=READY pgs=8389 cs=0 l=1 rev1=1 rx=0x7ff980045400 tx=0x7ff98c003ef0).stop 2021-03-09T10:44:21.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980025f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:21.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980025f90 unknown :-1 s=CLOSED pgs=8389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:21.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:21.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:21.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:21.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:21.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004730 con 0x7ff99c0adc20 2021-03-09T10:44:21.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:21.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026620 con 0x7ff99c0613a0 2021-03-09T10:44:21.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800175e0 con 0x7ff980051540 2021-03-09T10:44:21.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.309+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:21.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.313+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:21.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.313+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:21.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.313+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:21.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.313+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:21.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.313+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:21.320 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:21.313+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28405 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:21.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.313+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:21.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.313+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:44:21.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.313+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800480b0 con 0x7ff980051540 2021-03-09T10:44:21.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.317+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 secure :-1 s=READY pgs=8397 cs=0 l=1 rev1=1 rx=0x7ff98c079880 tx=0x7ff98c004ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:21.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.317+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c017330 con 0x7ff980051540 2021-03-09T10:44:21.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.321+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07cb80 con 0x7ff980051540 2021-03-09T10:44:21.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.325+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff980051540 2021-03-09T10:44:21.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.325+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:21.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.325+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:21.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:21.325+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:26.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:26.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:26.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 secure :-1 s=READY pgs=8397 cs=0 l=1 rev1=1 rx=0x7ff98c079880 tx=0x7ff98c004ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:26.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 secure :-1 s=READY pgs=8397 cs=0 l=1 rev1=1 rx=0x7ff98c079880 tx=0x7ff98c004ad0).stop 2021-03-09T10:44:26.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:26.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 unknown :-1 s=CLOSED pgs=8397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:26.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:26.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:26.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:26.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023710 con 0x7ff99c0adc20 2021-03-09T10:44:26.322 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011000 con 0x7ff99c0613a0 2021-03-09T10:44:26.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004730 con 0x7ff98000ac10 2021-03-09T10:44:26.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:26.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:26.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:26.324 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:26.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:26.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:26.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:26.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:26.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:26.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:44:26.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.317+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800480b0 con 0x7ff98000ac10 2021-03-09T10:44:26.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.321+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 secure :-1 s=READY pgs=8407 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c08d0e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:26.327 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:26.321+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28442 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:26.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.321+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff98000ac10 2021-03-09T10:44:26.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.321+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff98000ac10 2021-03-09T10:44:26.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.325+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff98000ac10 2021-03-09T10:44:26.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.325+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:26.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.329+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:26.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:26.329+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:31.325 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:31.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:31.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 secure :-1 s=READY pgs=8407 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c08d0e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:31.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 secure :-1 s=READY pgs=8407 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c08d0e0).stop 2021-03-09T10:44:31.326 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:31.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 unknown :-1 s=CLOSED pgs=8407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:31.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:31.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:31.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:31.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004730 con 0x7ff99c0adc20 2021-03-09T10:44:31.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046240 con 0x7ff99c0613a0 2021-03-09T10:44:31.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000cb50 con 0x7ff980051540 2021-03-09T10:44:31.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:31.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:31.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:31.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:31.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.321+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:31.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.325+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:31.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.325+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:31.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.325+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:31.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.325+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:31.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.325+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:44:31.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.325+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980003830 con 0x7ff980051540 2021-03-09T10:44:31.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.325+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 secure :-1 s=READY pgs=8415 cs=0 l=1 rev1=1 rx=0x7ff9940b1790 tx=0x7ff994009df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:31.333 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:31.325+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28479 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:31.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.325+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980051540 2021-03-09T10:44:31.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.325+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994014120 con 0x7ff980051540 2021-03-09T10:44:31.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.333+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:44:31.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.333+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:31.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.333+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:31.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:31.333+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:34.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:44:34.420 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:44:36.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:36.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:36.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 secure :-1 s=READY pgs=8415 cs=0 l=1 rev1=1 rx=0x7ff9940b1790 tx=0x7ff994009df0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:36.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 secure :-1 s=READY pgs=8415 cs=0 l=1 rev1=1 rx=0x7ff9940b1790 tx=0x7ff994009df0).stop 2021-03-09T10:44:36.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:36.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047390 unknown :-1 s=CLOSED pgs=8415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:36.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:36.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:36.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:36.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004730 con 0x7ff99c0adc20 2021-03-09T10:44:36.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046240 con 0x7ff99c0613a0 2021-03-09T10:44:36.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.325+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980012d40 con 0x7ff98000ac10 2021-03-09T10:44:36.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:36.334 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:36.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:36.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:36.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:36.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:36.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:36.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:36.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:36.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:44:36.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800480b0 con 0x7ff98000ac10 2021-03-09T10:44:36.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.329+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 secure :-1 s=READY pgs=8426 cs=0 l=1 rev1=1 rx=0x7ff98c018040 tx=0x7ff98c00baf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:36.337 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:36.329+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28516 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:36.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.333+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:44:36.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.333+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:44:36.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.337+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:44:36.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.337+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:36.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.337+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:36.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:36.337+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:41.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:41.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:41.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 secure :-1 s=READY pgs=8426 cs=0 l=1 rev1=1 rx=0x7ff98c018040 tx=0x7ff98c00baf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:41.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 secure :-1 s=READY pgs=8426 cs=0 l=1 rev1=1 rx=0x7ff98c018040 tx=0x7ff98c00baf0).stop 2021-03-09T10:44:41.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:41.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 unknown :-1 s=CLOSED pgs=8426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:41.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:41.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:41.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:41.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013e80 con 0x7ff99c0adc20 2021-03-09T10:44:41.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009bc0 con 0x7ff99c0613a0 2021-03-09T10:44:41.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004bcf0 con 0x7ff980051540 2021-03-09T10:44:41.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:41.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:41.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:41.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:41.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:41.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:41.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:41.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:41.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:41.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:44:41.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.333+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980003830 con 0x7ff980051540 2021-03-09T10:44:41.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.337+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047340 secure :-1 s=READY pgs=8438 cs=0 l=1 rev1=1 rx=0x7ff994060310 tx=0x7ff994014190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:41.343 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:41.337+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28553 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:41.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.341+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409be30 con 0x7ff980051540 2021-03-09T10:44:41.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.341+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095db0 con 0x7ff980051540 2021-03-09T10:44:41.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.341+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:44:41.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.341+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:41.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.345+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:41.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:41.345+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:46.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:46.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:46.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047340 secure :-1 s=READY pgs=8438 cs=0 l=1 rev1=1 rx=0x7ff994060310 tx=0x7ff994014190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:46.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047340 secure :-1 s=READY pgs=8438 cs=0 l=1 rev1=1 rx=0x7ff994060310 tx=0x7ff994014190).stop 2021-03-09T10:44:46.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980047340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:46.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980047340 unknown :-1 s=CLOSED pgs=8438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:46.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:46.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:46.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:46.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e10 con 0x7ff99c0adc20 2021-03-09T10:44:46.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff99c0613a0 2021-03-09T10:44:46.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023950 con 0x7ff98000ac10 2021-03-09T10:44:46.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:46.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:46.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:46.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:46.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.337+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:46.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.341+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:46.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.341+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:46.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.341+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:46.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.341+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:46.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.341+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800478a0 con 0x7ff98000ac10 2021-03-09T10:44:46.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.341+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980003830 con 0x7ff98000ac10 2021-03-09T10:44:46.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.341+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023460 secure :-1 s=READY pgs=8447 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99409bf10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:46.347 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:46.341+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28590 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:46.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.345+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095e80 con 0x7ff98000ac10 2021-03-09T10:44:46.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.349+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095e80 con 0x7ff98000ac10 2021-03-09T10:44:46.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.349+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:44:46.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.349+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:46.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.349+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:46.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:46.349+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:51.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:51.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:51.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023460 secure :-1 s=READY pgs=8447 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99409bf10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:51.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023460 secure :-1 s=READY pgs=8447 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99409bf10).stop 2021-03-09T10:44:51.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980023460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:51.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980023460 unknown :-1 s=CLOSED pgs=8447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:51.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:51.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:51.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:51.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e10 con 0x7ff99c0adc20 2021-03-09T10:44:51.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800039d0 con 0x7ff99c0613a0 2021-03-09T10:44:51.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:44:51.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:51.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:51.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:51.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:51.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.341+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:51.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.345+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:51.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.345+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:51.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.345+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:51.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.345+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:51.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.345+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020aa0 con 0x7ff980051540 2021-03-09T10:44:51.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.345+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800478a0 con 0x7ff980051540 2021-03-09T10:44:51.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.345+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8457 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff994060230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:51.353 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:51.345+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28627 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:51.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.349+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994084360 con 0x7ff980051540 2021-03-09T10:44:51.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.349+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994084180 con 0x7ff980051540 2021-03-09T10:44:51.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.353+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:44:51.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.353+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:51.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.353+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:51.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:51.353+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:44:56.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.345+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:44:56.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:56.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8457 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff994060230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:56.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8457 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff994060230).stop 2021-03-09T10:44:56.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:56.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=CLOSED pgs=8457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:56.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:56.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:56.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:56.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff99c0adc20 2021-03-09T10:44:56.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff99c0613a0 2021-03-09T10:44:56.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e10 con 0x7ff98000ac10 2021-03-09T10:44:56.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:44:56.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:44:56.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:56.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:56.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:56.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:56.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:56.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:56.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:56.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800039d0 con 0x7ff98000ac10 2021-03-09T10:44:56.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020aa0 con 0x7ff98000ac10 2021-03-09T10:44:56.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.349+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8469 cs=0 l=1 rev1=1 rx=0x7ff994084520 tx=0x7ff9940841f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:56.359 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:44:56.353+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28663 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:44:56.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.353+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940ab080 con 0x7ff98000ac10 2021-03-09T10:44:56.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.361+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940ab080 con 0x7ff98000ac10 2021-03-09T10:44:56.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.361+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:44:56.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.361+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:56.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.361+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:56.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:44:56.361+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:01.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:01.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:01.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8469 cs=0 l=1 rev1=1 rx=0x7ff994084520 tx=0x7ff9940841f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:01.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8469 cs=0 l=1 rev1=1 rx=0x7ff994084520 tx=0x7ff9940841f0).stop 2021-03-09T10:45:01.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:01.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=CLOSED pgs=8469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:01.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:01.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:01.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:01.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004be60 con 0x7ff99c0adc20 2021-03-09T10:45:01.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e10 con 0x7ff99c0613a0 2021-03-09T10:45:01.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff980051540 2021-03-09T10:45:01.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:01.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:01.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:01.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:01.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:01.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:01.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:01.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:01.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:01.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:45:01.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.352+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800480b0 con 0x7ff980051540 2021-03-09T10:45:01.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.356+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8479 cs=0 l=1 rev1=1 rx=0x7ff98c04c650 tx=0x7ff98c00ba80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:01.365 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:01.356+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28701 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:01.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.356+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980051540 2021-03-09T10:45:01.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.356+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:45:01.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.360+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:45:01.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.360+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:01.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.364+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:01.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:01.364+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:04.582 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:45:04.585 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:45:06.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:45:06.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:06.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8479 cs=0 l=1 rev1=1 rx=0x7ff98c04c650 tx=0x7ff98c00ba80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:06.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8479 cs=0 l=1 rev1=1 rx=0x7ff98c04c650 tx=0x7ff98c00ba80).stop 2021-03-09T10:45:06.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:06.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=CLOSED pgs=8479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:06.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:06.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:06.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:06.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800570e0 con 0x7ff99c0adc20 2021-03-09T10:45:06.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e10 con 0x7ff99c0613a0 2021-03-09T10:45:06.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff98000ac10 2021-03-09T10:45:06.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:06.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:06.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:06.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:06.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.356+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:06.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.360+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:06.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.360+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:06.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.360+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:06.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.360+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:06.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.360+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980048250 con 0x7ff98000ac10 2021-03-09T10:45:06.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.360+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800039d0 con 0x7ff98000ac10 2021-03-09T10:45:06.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.360+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8491 cs=0 l=1 rev1=1 rx=0x7ff994021040 tx=0x7ff994009d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:06.368 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:06.360+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28737 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:06.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.364+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405e1e0 con 0x7ff98000ac10 2021-03-09T10:45:06.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.364+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940011f0 con 0x7ff98000ac10 2021-03-09T10:45:06.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.368+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:45:06.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.368+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:06.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.368+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:06.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:06.368+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:11.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:45:11.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:11.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8491 cs=0 l=1 rev1=1 rx=0x7ff994021040 tx=0x7ff994009d80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:11.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8491 cs=0 l=1 rev1=1 rx=0x7ff994021040 tx=0x7ff994009d80).stop 2021-03-09T10:45:11.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:11.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=CLOSED pgs=8491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:11.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:11.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:11.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:11.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:11.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800570e0 con 0x7ff99c0adc20 2021-03-09T10:45:11.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:11.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e10 con 0x7ff99c0613a0 2021-03-09T10:45:11.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.360+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff980051540 2021-03-09T10:45:11.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.364+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:11.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.364+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:11.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.364+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:11.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.364+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:11.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.364+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:11.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.364+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:11.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.364+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:11.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.364+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004be60 con 0x7ff980051540 2021-03-09T10:45:11.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.364+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:45:11.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.364+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8501 cs=0 l=1 rev1=1 rx=0x7ff98c052520 tx=0x7ff98c052440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:11.374 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:11.368+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28775 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:11.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.368+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:45:11.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.368+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980051540 2021-03-09T10:45:11.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.376+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980051540 2021-03-09T10:45:11.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.376+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:11.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.376+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:11.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:11.376+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:16.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:45:16.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:16.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8501 cs=0 l=1 rev1=1 rx=0x7ff98c052520 tx=0x7ff98c052440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:16.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8501 cs=0 l=1 rev1=1 rx=0x7ff98c052520 tx=0x7ff98c052440).stop 2021-03-09T10:45:16.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:16.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=CLOSED pgs=8501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:16.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:16.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:16.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:16.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800570e0 con 0x7ff99c0adc20 2021-03-09T10:45:16.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e10 con 0x7ff99c0613a0 2021-03-09T10:45:16.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff98000ac10 2021-03-09T10:45:16.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:16.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:16.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:16.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:16.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.368+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:16.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.372+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:16.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.372+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:16.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.372+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:16.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.372+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:16.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.372+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800039d0 con 0x7ff98000ac10 2021-03-09T10:45:16.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.372+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004be60 con 0x7ff98000ac10 2021-03-09T10:45:16.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.372+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8512 cs=0 l=1 rev1=1 rx=0x7ff98c0852e0 tx=0x7ff98c008b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:16.385 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:16.372+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28811 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:16.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.372+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c020cf0 con 0x7ff98000ac10 2021-03-09T10:45:16.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.376+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c009540 con 0x7ff98000ac10 2021-03-09T10:45:16.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.376+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:45:16.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.376+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:16.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.380+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:16.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:16.380+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:21.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:45:21.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:21.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8512 cs=0 l=1 rev1=1 rx=0x7ff98c0852e0 tx=0x7ff98c008b00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:21.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8512 cs=0 l=1 rev1=1 rx=0x7ff98c0852e0 tx=0x7ff98c008b00).stop 2021-03-09T10:45:21.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:21.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=CLOSED pgs=8512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:21.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:21.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:21.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:21.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e10 con 0x7ff99c0adc20 2021-03-09T10:45:21.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff99c0613a0 2021-03-09T10:45:21.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004be60 con 0x7ff980051540 2021-03-09T10:45:21.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:21.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:21.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:21.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.372+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:21.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.376+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:21.383 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:21.376+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28849 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:21.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.380+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:21.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.380+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:21.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.380+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:21.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.380+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:21.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.380+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:45:21.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.380+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980048250 con 0x7ff980051540 2021-03-09T10:45:21.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.380+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8526 cs=0 l=1 rev1=1 rx=0x7ff99408c5f0 tx=0x7ff994060230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:21.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.384+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409dee0 con 0x7ff980051540 2021-03-09T10:45:21.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.388+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409dee0 con 0x7ff980051540 2021-03-09T10:45:21.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.388+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:45:21.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.388+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:21.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.388+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:21.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:21.388+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:26.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:26.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:26.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8526 cs=0 l=1 rev1=1 rx=0x7ff99408c5f0 tx=0x7ff994060230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:26.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8526 cs=0 l=1 rev1=1 rx=0x7ff99408c5f0 tx=0x7ff994060230).stop 2021-03-09T10:45:26.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:26.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=CLOSED pgs=8526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:26.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:26.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:26.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:26.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d1a0 con 0x7ff99c0adc20 2021-03-09T10:45:26.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e10 con 0x7ff99c0613a0 2021-03-09T10:45:26.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff98000ac10 2021-03-09T10:45:26.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:26.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:26.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:26.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:26.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:26.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:26.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:26.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.380+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:26.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.384+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:26.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.384+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004be60 con 0x7ff98000ac10 2021-03-09T10:45:26.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.384+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:45:26.390 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:26.384+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28885 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:26.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.384+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8537 cs=0 l=1 rev1=1 rx=0x7ff99405f380 tx=0x7ff994034b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:26.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.384+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095d70 con 0x7ff98000ac10 2021-03-09T10:45:26.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.388+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:45:26.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.392+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994095d70 con 0x7ff98000ac10 2021-03-09T10:45:26.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.392+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:26.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.392+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:26.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:26.392+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:31.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:31.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:31.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8537 cs=0 l=1 rev1=1 rx=0x7ff99405f380 tx=0x7ff994034b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:31.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 secure :-1 s=READY pgs=8537 cs=0 l=1 rev1=1 rx=0x7ff99405f380 tx=0x7ff994034b80).stop 2021-03-09T10:45:31.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980047340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:31.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980047340 unknown :-1 s=CLOSED pgs=8537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:31.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:31.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:31.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:31.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800094f0 con 0x7ff99c0adc20 2021-03-09T10:45:31.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020e10 con 0x7ff99c0613a0 2021-03-09T10:45:31.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff980051540 2021-03-09T10:45:31.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:31.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:31.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:31.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:31.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.384+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:31.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.388+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:31.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.388+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:31.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.388+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:31.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.388+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:31.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.388+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980048250 con 0x7ff980051540 2021-03-09T10:45:31.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.388+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004be60 con 0x7ff980051540 2021-03-09T10:45:31.396 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:31.388+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28923 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:31.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.388+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8545 cs=0 l=1 rev1=1 rx=0x7ff99409c040 tx=0x7ff99402afd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:31.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.388+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99402ad40 con 0x7ff980051540 2021-03-09T10:45:31.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.392+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99402ad40 con 0x7ff980051540 2021-03-09T10:45:31.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.396+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:45:31.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.396+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:31.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.396+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:31.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:31.396+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:34.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:45:34.627 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:45:36.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:45:36.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:36.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8545 cs=0 l=1 rev1=1 rx=0x7ff99409c040 tx=0x7ff99402afd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:36.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 secure :-1 s=READY pgs=8545 cs=0 l=1 rev1=1 rx=0x7ff99409c040 tx=0x7ff99402afd0).stop 2021-03-09T10:45:36.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:36.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026590 unknown :-1 s=CLOSED pgs=8545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:36.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980026a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:36.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:36.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:36.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff99c0adc20 2021-03-09T10:45:36.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026f70 con 0x7ff99c0613a0 2021-03-09T10:45:36.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800094a0 con 0x7ff98000ac10 2021-03-09T10:45:36.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:36.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:36.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:36.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980026a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:36.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.388+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:36.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.392+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:36.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.392+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:36.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.392+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:36.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.392+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:36.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.392+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008d1a0 con 0x7ff98000ac10 2021-03-09T10:45:36.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.392+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980048250 con 0x7ff98000ac10 2021-03-09T10:45:36.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.392+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980026a80 secure :-1 s=READY pgs=8553 cs=0 l=1 rev1=1 rx=0x7ff99405e550 tx=0x7ff9940816e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:36.401 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:36.392+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28959 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:36.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.396+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99402ad40 con 0x7ff98000ac10 2021-03-09T10:45:36.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.396+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99402ad40 con 0x7ff98000ac10 2021-03-09T10:45:36.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.400+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:45:36.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.400+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:36.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.400+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:36.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:36.400+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:41.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980026a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:41.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980026a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:41.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980026a80 secure :-1 s=READY pgs=8553 cs=0 l=1 rev1=1 rx=0x7ff99405e550 tx=0x7ff9940816e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:41.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980026a80 secure :-1 s=READY pgs=8553 cs=0 l=1 rev1=1 rx=0x7ff99405e550 tx=0x7ff9940816e0).stop 2021-03-09T10:45:41.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980026a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:41.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980026a80 unknown :-1 s=CLOSED pgs=8553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:41.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:41.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:41.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:41.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056ee0 con 0x7ff99c0adc20 2021-03-09T10:45:41.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff99c0613a0 2021-03-09T10:45:41.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020aa0 con 0x7ff980051540 2021-03-09T10:45:41.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:41.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:41.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:41.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:41.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:41.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:41.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:41.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:41.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:41.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047340 con 0x7ff980051540 2021-03-09T10:45:41.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008d1a0 con 0x7ff980051540 2021-03-09T10:45:41.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.396+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026540 secure :-1 s=READY pgs=8566 cs=0 l=1 rev1=1 rx=0x7ff99400c310 tx=0x7ff99405f310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:41.406 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:41.400+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28997 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:41.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.400+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994001e90 con 0x7ff980051540 2021-03-09T10:45:41.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.400+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405e1e0 con 0x7ff980051540 2021-03-09T10:45:41.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.404+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:45:41.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.404+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:41.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.404+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:41.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:41.404+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:46.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:46.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:46.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026540 secure :-1 s=READY pgs=8566 cs=0 l=1 rev1=1 rx=0x7ff99400c310 tx=0x7ff99405f310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:46.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026540 secure :-1 s=READY pgs=8566 cs=0 l=1 rev1=1 rx=0x7ff99400c310 tx=0x7ff99405f310).stop 2021-03-09T10:45:46.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:46.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026540 unknown :-1 s=CLOSED pgs=8566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:46.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:46.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:46.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:46.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056da0 con 0x7ff99c0adc20 2021-03-09T10:45:46.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800479c0 con 0x7ff99c0613a0 2021-03-09T10:45:46.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b4d0 con 0x7ff98000ac10 2021-03-09T10:45:46.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:46.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:46.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:46.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:46.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.400+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:46.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.404+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:46.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.404+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:46.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.404+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:46.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.404+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:46.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.404+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004b610 con 0x7ff98000ac10 2021-03-09T10:45:46.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.404+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047340 con 0x7ff98000ac10 2021-03-09T10:45:46.410 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:46.404+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29033 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:46.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.404+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800474d0 secure :-1 s=READY pgs=8577 cs=0 l=1 rev1=1 rx=0x7ff99405f380 tx=0x7ff99406dea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:46.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.408+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994001e90 con 0x7ff98000ac10 2021-03-09T10:45:46.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.408+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405e1e0 con 0x7ff98000ac10 2021-03-09T10:45:46.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.412+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:45:46.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.412+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:46.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.412+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:46.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:46.412+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:51.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:51.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:51.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800474d0 secure :-1 s=READY pgs=8577 cs=0 l=1 rev1=1 rx=0x7ff99405f380 tx=0x7ff99406dea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:51.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800474d0 secure :-1 s=READY pgs=8577 cs=0 l=1 rev1=1 rx=0x7ff99405f380 tx=0x7ff99406dea0).stop 2021-03-09T10:45:51.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800474d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:51.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800474d0 unknown :-1 s=CLOSED pgs=8577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:51.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:51.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:51.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:51.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b4d0 con 0x7ff99c0adc20 2021-03-09T10:45:51.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048720 con 0x7ff99c0613a0 2021-03-09T10:45:51.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047340 con 0x7ff980051540 2021-03-09T10:45:51.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:51.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:51.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:51.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:51.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.404+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:51.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.408+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:51.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.408+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:51.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.408+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:51.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.408+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:51.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.408+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:45:51.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.408+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800039d0 con 0x7ff980051540 2021-03-09T10:45:51.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.408+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 secure :-1 s=READY pgs=8586 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c079340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:51.416 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:51.408+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29072 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:51.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.412+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980051540 2021-03-09T10:45:51.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.412+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:45:51.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.416+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:45:51.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.416+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:51.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.416+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:51.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:51.416+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:45:56.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:56.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:56.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 secure :-1 s=READY pgs=8586 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c079340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:56.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 secure :-1 s=READY pgs=8586 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c079340).stop 2021-03-09T10:45:56.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:56.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 unknown :-1 s=CLOSED pgs=8586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:56.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:56.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:56.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:56.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b4d0 con 0x7ff99c0adc20 2021-03-09T10:45:56.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800063e0 con 0x7ff99c0613a0 2021-03-09T10:45:56.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056f80 con 0x7ff98000ac10 2021-03-09T10:45:56.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:45:56.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:56.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:45:56.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.408+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:56.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.412+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:56.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.412+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:56.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.412+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:56.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.412+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:56.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.412+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:56.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.412+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800039d0 con 0x7ff98000ac10 2021-03-09T10:45:56.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.412+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:45:56.421 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:45:56.412+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29108 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:45:56.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.416+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 secure :-1 s=READY pgs=8604 cs=0 l=1 rev1=1 rx=0x7ff98c02d050 tx=0x7ff98c084e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:56.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.420+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c008ca0 con 0x7ff98000ac10 2021-03-09T10:45:56.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:45:56.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c008c50 con 0x7ff98000ac10 2021-03-09T10:45:56.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.424+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:56.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.424+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:56.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:45:56.424+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:01.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:01.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:01.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 secure :-1 s=READY pgs=8604 cs=0 l=1 rev1=1 rx=0x7ff98c02d050 tx=0x7ff98c084e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:01.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 secure :-1 s=READY pgs=8604 cs=0 l=1 rev1=1 rx=0x7ff98c02d050 tx=0x7ff98c084e80).stop 2021-03-09T10:46:01.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:01.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096da0 unknown :-1 s=CLOSED pgs=8604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:01.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:01.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:01.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:01.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b4d0 con 0x7ff99c0adc20 2021-03-09T10:46:01.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056f80 con 0x7ff99c0613a0 2021-03-09T10:46:01.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800409a0 con 0x7ff980051540 2021-03-09T10:46:01.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:01.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:01.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:01.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:01.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.416+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:01.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:01.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:01.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:01.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:01.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:46:01.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004b610 con 0x7ff980051540 2021-03-09T10:46:01.427 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:01.420+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29147 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:01.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 secure :-1 s=READY pgs=8609 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff994078040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:01.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99406df10 con 0x7ff980051540 2021-03-09T10:46:01.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980051540 2021-03-09T10:46:01.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.428+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:46:01.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.428+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:01.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.428+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:01.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:01.428+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:04.671 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:46:04.674 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:46:06.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:06.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:06.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 secure :-1 s=READY pgs=8609 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff994078040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:06.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 secure :-1 s=READY pgs=8609 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff994078040).stop 2021-03-09T10:46:06.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980026bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:06.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980026bf0 unknown :-1 s=CLOSED pgs=8609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:06.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:06.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:06.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:06.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014720 con 0x7ff99c0adc20 2021-03-09T10:46:06.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056f80 con 0x7ff99c0613a0 2021-03-09T10:46:06.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092fb0 con 0x7ff98000ac10 2021-03-09T10:46:06.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:06.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:06.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:06.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:06.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.420+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:06.429 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.424+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:06.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.424+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:06.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.424+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:06.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.424+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:06.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.424+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800264c0 con 0x7ff98000ac10 2021-03-09T10:46:06.430 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.424+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:46:06.431 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:06.424+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29183 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:06.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.424+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015ac0 secure :-1 s=READY pgs=8623 cs=0 l=1 rev1=1 rx=0x7ff994014930 tx=0x7ff9940147e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:06.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff98000ac10 2021-03-09T10:46:06.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:46:06.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994095d70 con 0x7ff98000ac10 2021-03-09T10:46:06.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.436+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:06.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.436+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:06.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:06.436+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:11.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:46:11.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:11.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015ac0 secure :-1 s=READY pgs=8623 cs=0 l=1 rev1=1 rx=0x7ff994014930 tx=0x7ff9940147e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:11.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015ac0 secure :-1 s=READY pgs=8623 cs=0 l=1 rev1=1 rx=0x7ff994014930 tx=0x7ff9940147e0).stop 2021-03-09T10:46:11.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:11.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015ac0 unknown :-1 s=CLOSED pgs=8623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:11.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:11.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:11.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:11.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013e20 con 0x7ff99c0adc20 2021-03-09T10:46:11.433 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014720 con 0x7ff99c0613a0 2021-03-09T10:46:11.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056f80 con 0x7ff980051540 2021-03-09T10:46:11.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.424+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:11.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:11.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:11.434 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:11.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:11.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:11.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:11.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:11.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:11.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:46:11.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800264c0 con 0x7ff980051540 2021-03-09T10:46:11.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.428+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015fb0 secure :-1 s=READY pgs=8631 cs=0 l=1 rev1=1 rx=0x7ff9940aadf0 tx=0x7ff9940aaf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:11.438 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:11.432+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29222 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:11.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff980051540 2021-03-09T10:46:11.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.432+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994049700 con 0x7ff980051540 2021-03-09T10:46:11.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.436+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:46:11.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.436+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:11.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.436+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:11.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:11.436+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:16.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:46:16.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:16.436 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015fb0 secure :-1 s=READY pgs=8631 cs=0 l=1 rev1=1 rx=0x7ff9940aadf0 tx=0x7ff9940aaf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:16.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015fb0 secure :-1 s=READY pgs=8631 cs=0 l=1 rev1=1 rx=0x7ff9940aadf0 tx=0x7ff9940aaf90).stop 2021-03-09T10:46:16.437 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:16.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015fb0 unknown :-1 s=CLOSED pgs=8631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:16.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:16.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:16.438 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:16.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005ee0 con 0x7ff99c0adc20 2021-03-09T10:46:16.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014720 con 0x7ff99c0613a0 2021-03-09T10:46:16.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056f80 con 0x7ff98000ac10 2021-03-09T10:46:16.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:16.439 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:16.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:16.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:16.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.431+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:16.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.435+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:16.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.435+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:16.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.435+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:16.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.435+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:16.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.435+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013e20 con 0x7ff98000ac10 2021-03-09T10:46:16.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.435+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:46:16.443 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:16.435+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29258 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:16.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.435+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800158d0 secure :-1 s=READY pgs=8642 cs=0 l=1 rev1=1 rx=0x7ff994001d60 tx=0x7ff99405f240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:16.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.435+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994012ec0 con 0x7ff98000ac10 2021-03-09T10:46:16.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.439+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:46:16.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.443+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:46:16.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.443+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:16.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.443+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:16.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:16.443+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:21.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:46:21.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:21.442 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800158d0 secure :-1 s=READY pgs=8642 cs=0 l=1 rev1=1 rx=0x7ff994001d60 tx=0x7ff99405f240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:21.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800158d0 secure :-1 s=READY pgs=8642 cs=0 l=1 rev1=1 rx=0x7ff994001d60 tx=0x7ff99405f240).stop 2021-03-09T10:46:21.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:21.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800158d0 unknown :-1 s=CLOSED pgs=8642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:21.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:21.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:21.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:21.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014720 con 0x7ff99c0adc20 2021-03-09T10:46:21.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056f80 con 0x7ff99c0613a0 2021-03-09T10:46:21.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a470 con 0x7ff980051540 2021-03-09T10:46:21.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:21.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:21.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:21.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:21.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.435+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:21.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.439+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:21.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.439+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:21.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.439+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:21.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.439+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:21.447 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.439+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980028e30 con 0x7ff980051540 2021-03-09T10:46:21.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.439+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013e20 con 0x7ff980051540 2021-03-09T10:46:21.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.439+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015dc0 secure :-1 s=READY pgs=8649 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994020220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:21.448 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:21.443+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29297 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:21.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.443+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020ec0 con 0x7ff980051540 2021-03-09T10:46:21.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.443+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:46:21.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.447+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994095d70 con 0x7ff980051540 2021-03-09T10:46:21.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.447+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:21.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.447+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:21.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:21.447+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:26.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:26.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:26.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015dc0 secure :-1 s=READY pgs=8649 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994020220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:26.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015dc0 secure :-1 s=READY pgs=8649 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994020220).stop 2021-03-09T10:46:26.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:26.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015dc0 unknown :-1 s=CLOSED pgs=8649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:26.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:26.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:26.450 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:26.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014720 con 0x7ff99c0adc20 2021-03-09T10:46:26.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800264c0 con 0x7ff99c0613a0 2021-03-09T10:46:26.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:26.451 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d7a0 con 0x7ff98000ac10 2021-03-09T10:46:26.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:26.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:26.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:26.452 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:26.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:26.453 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:26.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:26.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:26.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002d910 con 0x7ff98000ac10 2021-03-09T10:46:26.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800039d0 con 0x7ff98000ac10 2021-03-09T10:46:26.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.443+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015830 secure :-1 s=READY pgs=8661 cs=0 l=1 rev1=1 rx=0x7ff98c052410 tx=0x7ff98c09ffc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:26.455 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:26.447+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29333 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:26.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.447+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079760 con 0x7ff98000ac10 2021-03-09T10:46:26.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.447+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a0130 con 0x7ff98000ac10 2021-03-09T10:46:26.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.451+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:46:26.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.451+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:26.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.451+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:26.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:26.451+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:31.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:31.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:31.454 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015830 secure :-1 s=READY pgs=8661 cs=0 l=1 rev1=1 rx=0x7ff98c052410 tx=0x7ff98c09ffc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:31.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015830 secure :-1 s=READY pgs=8661 cs=0 l=1 rev1=1 rx=0x7ff98c052410 tx=0x7ff98c09ffc0).stop 2021-03-09T10:46:31.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:31.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015830 unknown :-1 s=CLOSED pgs=8661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:31.455 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:31.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:31.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:31.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d1d0 con 0x7ff99c0adc20 2021-03-09T10:46:31.456 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014720 con 0x7ff99c0613a0 2021-03-09T10:46:31.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006a90 con 0x7ff980051540 2021-03-09T10:46:31.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:31.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:31.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:31.457 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:31.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:31.458 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:31.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:31.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:31.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:31.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800039d0 con 0x7ff980051540 2021-03-09T10:46:31.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.447+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980028e30 con 0x7ff980051540 2021-03-09T10:46:31.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.451+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015d20 secure :-1 s=READY pgs=8672 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994014bd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:31.461 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:31.451+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29372 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:31.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.451+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409cea0 con 0x7ff980051540 2021-03-09T10:46:31.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.451+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069530 con 0x7ff980051540 2021-03-09T10:46:31.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.455+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:46:31.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.455+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:31.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.455+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:31.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:31.455+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:34.755 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:46:34.758 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:46:36.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:36.459 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:36.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015d20 secure :-1 s=READY pgs=8672 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994014bd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:36.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015d20 secure :-1 s=READY pgs=8672 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994014bd0).stop 2021-03-09T10:46:36.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:36.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015d20 unknown :-1 s=CLOSED pgs=8672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:36.460 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:36.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:36.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:36.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d1d0 con 0x7ff99c0adc20 2021-03-09T10:46:36.461 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014720 con 0x7ff99c0613a0 2021-03-09T10:46:36.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c210 con 0x7ff98000ac10 2021-03-09T10:46:36.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:36.462 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:36.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:36.463 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:36.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:36.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:36.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:36.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:36.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:36.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980028630 con 0x7ff98000ac10 2021-03-09T10:46:36.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800039d0 con 0x7ff98000ac10 2021-03-09T10:46:36.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.455+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8682 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99400c1c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:36.466 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:36.459+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29408 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:36.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b11c0 con 0x7ff98000ac10 2021-03-09T10:46:36.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069350 con 0x7ff98000ac10 2021-03-09T10:46:36.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:46:36.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.467+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:36.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.467+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:36.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:36.467+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:41.464 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:46:41.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:41.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8682 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99400c1c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:41.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8682 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff99400c1c0).stop 2021-03-09T10:46:41.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:41.465 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=CLOSED pgs=8682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:41.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:41.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800142f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:41.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:41.466 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006a90 con 0x7ff99c0adc20 2021-03-09T10:46:41.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800115a0 con 0x7ff99c0613a0 2021-03-09T10:46:41.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c1e0 con 0x7ff980051540 2021-03-09T10:46:41.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:41.467 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:41.468 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:41.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:41.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800142f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:41.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:41.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:41.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:41.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:41.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:46:41.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.459+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002d910 con 0x7ff980051540 2021-03-09T10:46:41.471 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:41.463+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29446 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:41.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.463+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800142f0 secure :-1 s=READY pgs=8694 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c0a01a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:41.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980051540 2021-03-09T10:46:41.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:46:41.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:46:41.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.467+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:41.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.471+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:41.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:41.471+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:46.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800142f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:46.469 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800142f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:46.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800142f0 secure :-1 s=READY pgs=8694 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c0a01a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:46.470 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800142f0 secure :-1 s=READY pgs=8694 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c0a01a0).stop 2021-03-09T10:46:46.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800142f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:46.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800142f0 unknown :-1 s=CLOSED pgs=8694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:46.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:46.471 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:46.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:46.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:46.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006a90 con 0x7ff99c0adc20 2021-03-09T10:46:46.472 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:46.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800115a0 con 0x7ff99c0613a0 2021-03-09T10:46:46.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:46:46.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:46.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:46.473 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.463+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:46.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.467+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:46.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.467+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:46.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.467+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:46.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.467+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:46.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.467+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800264c0 con 0x7ff98000ac10 2021-03-09T10:46:46.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.467+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002d910 con 0x7ff98000ac10 2021-03-09T10:46:46.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.467+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8703 cs=0 l=1 rev1=1 rx=0x7ff98c009ad0 tx=0x7ff98c08df00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:46.477 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:46.467+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29482 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:46.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.467+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08fbf0 con 0x7ff98000ac10 2021-03-09T10:46:46.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08fd50 con 0x7ff98000ac10 2021-03-09T10:46:46.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c08fbf0 con 0x7ff98000ac10 2021-03-09T10:46:46.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.475+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:46.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.475+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:46.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:46.475+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:51.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:51.475 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:51.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8703 cs=0 l=1 rev1=1 rx=0x7ff98c009ad0 tx=0x7ff98c08df00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:51.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8703 cs=0 l=1 rev1=1 rx=0x7ff98c009ad0 tx=0x7ff98c08df00).stop 2021-03-09T10:46:51.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:51.476 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=CLOSED pgs=8703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:51.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:51.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:51.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:51.477 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006a90 con 0x7ff99c0adc20 2021-03-09T10:46:51.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800115a0 con 0x7ff99c0613a0 2021-03-09T10:46:51.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800264c0 con 0x7ff980051540 2021-03-09T10:46:51.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:51.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:51.478 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:51.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:51.479 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:51.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:51.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:51.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:51.480 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:51.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:46:51.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980028630 con 0x7ff980051540 2021-03-09T10:46:51.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.471+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 secure :-1 s=READY pgs=8716 cs=0 l=1 rev1=1 rx=0x7ff994081db0 tx=0x7ff9940682d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:51.481 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:51.475+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29520 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:51.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:46:51.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069350 con 0x7ff980051540 2021-03-09T10:46:51.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.483+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:46:51.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.483+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:51.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.483+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:51.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:51.483+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:46:56.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:56.481 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:56.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 secure :-1 s=READY pgs=8716 cs=0 l=1 rev1=1 rx=0x7ff994081db0 tx=0x7ff9940682d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:56.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 secure :-1 s=READY pgs=8716 cs=0 l=1 rev1=1 rx=0x7ff994081db0 tx=0x7ff9940682d0).stop 2021-03-09T10:46:56.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:56.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 unknown :-1 s=CLOSED pgs=8716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:56.482 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:56.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:56.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:56.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006a90 con 0x7ff99c0adc20 2021-03-09T10:46:56.483 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011550 con 0x7ff99c0613a0 2021-03-09T10:46:56.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800149d0 con 0x7ff98000ac10 2021-03-09T10:46:56.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:46:56.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:56.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:46:56.484 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:56.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:56.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:56.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:56.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:56.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:56.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800264c0 con 0x7ff98000ac10 2021-03-09T10:46:56.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:46:56.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.475+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8725 cs=0 l=1 rev1=1 rx=0x7ff99402a340 tx=0x7ff994020220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:56.487 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:46:56.479+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29557 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:46:56.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b1d80 con 0x7ff98000ac10 2021-03-09T10:46:56.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069350 con 0x7ff98000ac10 2021-03-09T10:46:56.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.483+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:46:56.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.483+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:56.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.483+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:56.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:46:56.483+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:01.485 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:01.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:01.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8725 cs=0 l=1 rev1=1 rx=0x7ff99402a340 tx=0x7ff994020220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:01.486 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8725 cs=0 l=1 rev1=1 rx=0x7ff99402a340 tx=0x7ff994020220).stop 2021-03-09T10:47:01.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:01.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=CLOSED pgs=8725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:01.487 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:01.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:01.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:01.488 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980054b90 con 0x7ff99c0adc20 2021-03-09T10:47:01.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006a90 con 0x7ff99c0613a0 2021-03-09T10:47:01.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:47:01.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:01.489 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:01.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:01.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:01.490 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.479+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:01.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.483+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:01.491 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:01.483+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29593 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:01.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.483+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:01.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.483+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:01.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.483+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:01.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.483+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020aa0 con 0x7ff980051540 2021-03-09T10:47:01.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.483+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800264c0 con 0x7ff980051540 2021-03-09T10:47:01.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.487+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 secure :-1 s=READY pgs=8740 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff994095120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:01.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980051540 2021-03-09T10:47:01.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400c380 con 0x7ff980051540 2021-03-09T10:47:01.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:47:01.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.491+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:01.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.491+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:01.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:01.491+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:04.810 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:47:04.813 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:47:06.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:06.491 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:06.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 secure :-1 s=READY pgs=8740 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff994095120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:06.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 secure :-1 s=READY pgs=8740 cs=0 l=1 rev1=1 rx=0x7ff99400b440 tx=0x7ff994095120).stop 2021-03-09T10:47:06.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:06.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 unknown :-1 s=CLOSED pgs=8740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:06.492 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:06.493 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:06.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:06.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff99c0adc20 2021-03-09T10:47:06.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017340 con 0x7ff99c0613a0 2021-03-09T10:47:06.494 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006a90 con 0x7ff98000ac10 2021-03-09T10:47:06.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:06.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:06.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:06.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:06.495 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:06.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:06.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:06.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:06.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:06.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980054b40 con 0x7ff98000ac10 2021-03-09T10:47:06.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020aa0 con 0x7ff98000ac10 2021-03-09T10:47:06.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.487+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8747 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994029c40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:06.498 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:06.491+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29632 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:06.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400c380 con 0x7ff98000ac10 2021-03-09T10:47:06.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994029950 con 0x7ff98000ac10 2021-03-09T10:47:06.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:47:06.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.495+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:06.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.495+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:06.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:06.495+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:11.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:11.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:11.496 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8747 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994029c40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:11.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8747 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994029c40).stop 2021-03-09T10:47:11.497 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:11.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=CLOSED pgs=8747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:11.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:11.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:11.498 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:11.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006c90 con 0x7ff99c0adc20 2021-03-09T10:47:11.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff99c0613a0 2021-03-09T10:47:11.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020aa0 con 0x7ff980051540 2021-03-09T10:47:11.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:11.499 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:11.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:11.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:11.500 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:11.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:11.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:11.501 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:11.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:11.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800264c0 con 0x7ff980051540 2021-03-09T10:47:11.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.491+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002d910 con 0x7ff980051540 2021-03-09T10:47:11.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.495+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046380 secure :-1 s=READY pgs=8757 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0a0130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:11.504 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:11.495+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29668 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:11.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:47:11.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.495+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:47:11.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980051540 2021-03-09T10:47:11.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.499+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:11.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.503+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:11.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:11.503+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:16.502 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980046380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:16.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980046380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:16.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046380 secure :-1 s=READY pgs=8757 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0a0130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:16.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046380 secure :-1 s=READY pgs=8757 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c0a0130).stop 2021-03-09T10:47:16.503 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980046380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:16.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046380 unknown :-1 s=CLOSED pgs=8757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:16.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:16.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:16.504 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:16.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d8f0 con 0x7ff99c0adc20 2021-03-09T10:47:16.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002da30 con 0x7ff99c0613a0 2021-03-09T10:47:16.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015b20 con 0x7ff98000ac10 2021-03-09T10:47:16.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:16.505 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:16.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:16.506 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:16.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:16.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:16.507 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:16.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:16.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:16.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015c60 con 0x7ff98000ac10 2021-03-09T10:47:16.508 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.499+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980054b40 con 0x7ff98000ac10 2021-03-09T10:47:16.509 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:16.499+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29707 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:16.509 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.503+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8771 cs=0 l=1 rev1=1 rx=0x7ff99405ce40 tx=0x7ff994036960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:16.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:47:16.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.507+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:47:16.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.511+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:47:16.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.511+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:16.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.511+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:16.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:16.511+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:21.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:21.510 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:21.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8771 cs=0 l=1 rev1=1 rx=0x7ff99405ce40 tx=0x7ff994036960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:21.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8771 cs=0 l=1 rev1=1 rx=0x7ff99405ce40 tx=0x7ff994036960).stop 2021-03-09T10:47:21.511 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:21.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=CLOSED pgs=8771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:21.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:21.512 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:21.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:21.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015b20 con 0x7ff99c0adc20 2021-03-09T10:47:21.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff99c0613a0 2021-03-09T10:47:21.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d750 con 0x7ff980051540 2021-03-09T10:47:21.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:21.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:21.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:21.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:21.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:21.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:21.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:21.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:21.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:21.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002d890 con 0x7ff980051540 2021-03-09T10:47:21.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800264c0 con 0x7ff980051540 2021-03-09T10:47:21.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.503+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015df0 secure :-1 s=READY pgs=8777 cs=0 l=1 rev1=1 rx=0x7ff9800291c0 tx=0x7ff98c08ffb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:21.520 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:21.507+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29743 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:21.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.507+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c014650 con 0x7ff980051540 2021-03-09T10:47:21.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.507+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:47:21.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.511+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:47:21.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.511+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:21.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.511+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:21.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:21.511+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:26.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:26.513 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:26.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015df0 secure :-1 s=READY pgs=8777 cs=0 l=1 rev1=1 rx=0x7ff9800291c0 tx=0x7ff98c08ffb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:26.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015df0 secure :-1 s=READY pgs=8777 cs=0 l=1 rev1=1 rx=0x7ff9800291c0 tx=0x7ff98c08ffb0).stop 2021-03-09T10:47:26.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015df0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:26.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015df0 unknown :-1 s=CLOSED pgs=8777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:26.514 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:26.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:26.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:26.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015b20 con 0x7ff99c0adc20 2021-03-09T10:47:26.515 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d700 con 0x7ff99c0613a0 2021-03-09T10:47:26.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026490 con 0x7ff98000ac10 2021-03-09T10:47:26.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:26.516 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:26.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:26.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:26.517 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.507+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:26.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.511+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:26.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.511+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:26.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.511+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:26.519 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:26.511+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29782 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:26.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.511+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:26.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.511+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002da20 con 0x7ff98000ac10 2021-03-09T10:47:26.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.511+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015c60 con 0x7ff98000ac10 2021-03-09T10:47:26.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.515+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8794 cs=0 l=1 rev1=1 rx=0x7ff980009d40 tx=0x7ff994084f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:26.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.515+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff98000ac10 2021-03-09T10:47:26.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.515+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff98000ac10 2021-03-09T10:47:26.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.519+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:47:26.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.519+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:26.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.519+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:26.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:26.519+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:31.518 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:31.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:31.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8794 cs=0 l=1 rev1=1 rx=0x7ff980009d40 tx=0x7ff994084f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:31.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8794 cs=0 l=1 rev1=1 rx=0x7ff980009d40 tx=0x7ff994084f10).stop 2021-03-09T10:47:31.519 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:31.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=CLOSED pgs=8794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:31.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:31.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:31.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:31.520 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800292e0 con 0x7ff99c0adc20 2021-03-09T10:47:31.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048230 con 0x7ff99c0613a0 2021-03-09T10:47:31.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026490 con 0x7ff980051540 2021-03-09T10:47:31.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:31.521 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:31.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:31.522 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:31.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:31.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:31.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:31.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:31.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:31.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002d700 con 0x7ff980051540 2021-03-09T10:47:31.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002da20 con 0x7ff980051540 2021-03-09T10:47:31.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.514+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 secure :-1 s=READY pgs=8800 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400ca30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:31.525 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:31.514+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29818 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:31.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940aae20 con 0x7ff980051540 2021-03-09T10:47:31.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980051540 2021-03-09T10:47:31.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:47:31.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.522+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:31.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.522+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:31.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:31.522+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:34.873 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:47:34.876 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:47:36.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:36.523 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:36.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 secure :-1 s=READY pgs=8800 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400ca30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:36.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 secure :-1 s=READY pgs=8800 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400ca30).stop 2021-03-09T10:47:36.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980015b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:36.524 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980015b20 unknown :-1 s=CLOSED pgs=8800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:36.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:36.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:36.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:36.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048230 con 0x7ff99c0adc20 2021-03-09T10:47:36.525 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026470 con 0x7ff99c0613a0 2021-03-09T10:47:36.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980089210 con 0x7ff98000ac10 2021-03-09T10:47:36.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:36.526 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:36.527 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:36.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:36.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:36.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:36.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:36.528 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:36.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:36.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002da20 con 0x7ff98000ac10 2021-03-09T10:47:36.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.518+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002d700 con 0x7ff98000ac10 2021-03-09T10:47:36.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8810 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99407e4e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:36.530 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:36.522+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29857 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:36.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff98000ac10 2021-03-09T10:47:36.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940aae20 con 0x7ff98000ac10 2021-03-09T10:47:36.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.526+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:47:36.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.526+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:36.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.530+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:36.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:36.530+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:41.529 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:41.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:41.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8810 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99407e4e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:41.530 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8810 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99407e4e0).stop 2021-03-09T10:47:41.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:41.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=CLOSED pgs=8810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:41.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:41.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:41.531 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:41.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048230 con 0x7ff99c0adc20 2021-03-09T10:47:41.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff99c0613a0 2021-03-09T10:47:41.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015b20 con 0x7ff980051540 2021-03-09T10:47:41.532 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:41.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:41.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:41.533 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:41.534 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.522+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:41.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.526+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:41.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.526+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:41.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.526+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:41.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.526+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:41.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.526+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015c90 con 0x7ff980051540 2021-03-09T10:47:41.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.526+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002d890 con 0x7ff980051540 2021-03-09T10:47:41.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.526+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016fb0 secure :-1 s=READY pgs=8821 cs=0 l=1 rev1=1 rx=0x7ff98c0873e0 tx=0x7ff98c0a2f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:41.536 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:41.526+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29893 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:41.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:47:41.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980051540 2021-03-09T10:47:41.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:47:41.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.534+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:41.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.534+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:41.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:41.534+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:46.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:46.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:46.535 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016fb0 secure :-1 s=READY pgs=8821 cs=0 l=1 rev1=1 rx=0x7ff98c0873e0 tx=0x7ff98c0a2f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:46.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016fb0 secure :-1 s=READY pgs=8821 cs=0 l=1 rev1=1 rx=0x7ff98c0873e0 tx=0x7ff98c0a2f40).stop 2021-03-09T10:47:46.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:46.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016fb0 unknown :-1 s=CLOSED pgs=8821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:46.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:46.536 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:46.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:46.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b770 con 0x7ff99c0adc20 2021-03-09T10:47:46.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d1b0 con 0x7ff99c0613a0 2021-03-09T10:47:46.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800475e0 con 0x7ff98000ac10 2021-03-09T10:47:46.537 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:46.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:46.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:46.538 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:46.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:46.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:46.539 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:46.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:46.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:46.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002d890 con 0x7ff98000ac10 2021-03-09T10:47:46.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002da20 con 0x7ff98000ac10 2021-03-09T10:47:46.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.530+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8831 cs=0 l=1 rev1=1 rx=0x7ff99408e580 tx=0x7ff99407e470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:46.541 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:46.534+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29932 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:46.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:47:46.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994029350 con 0x7ff98000ac10 2021-03-09T10:47:46.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:47:46.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.538+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:46.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.538+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:46.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:46.538+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:51.540 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:51.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:51.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8831 cs=0 l=1 rev1=1 rx=0x7ff99408e580 tx=0x7ff99407e470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:51.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 secure :-1 s=READY pgs=8831 cs=0 l=1 rev1=1 rx=0x7ff99408e580 tx=0x7ff99407e470).stop 2021-03-09T10:47:51.541 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:51.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015630 unknown :-1 s=CLOSED pgs=8831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:51.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:51.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:51.542 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:51.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b540 con 0x7ff99c0adc20 2021-03-09T10:47:51.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006850 con 0x7ff99c0613a0 2021-03-09T10:47:51.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015b20 con 0x7ff980051540 2021-03-09T10:47:51.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:51.543 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:51.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:51.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:51.544 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.534+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:51.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.538+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:51.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.538+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:51.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.538+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:51.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.538+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:51.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.538+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:47:51.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.538+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002d890 con 0x7ff980051540 2021-03-09T10:47:51.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.538+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016f10 secure :-1 s=READY pgs=8842 cs=0 l=1 rev1=1 rx=0x7ff99405efc0 tx=0x7ff994029450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:51.547 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:51.538+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29968 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:51.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069350 con 0x7ff980051540 2021-03-09T10:47:51.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.538+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99402ee60 con 0x7ff980051540 2021-03-09T10:47:51.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:47:51.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.542+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:51.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.542+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:51.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:51.542+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:47:56.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:56.545 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:56.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016f10 secure :-1 s=READY pgs=8842 cs=0 l=1 rev1=1 rx=0x7ff99405efc0 tx=0x7ff994029450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:56.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016f10 secure :-1 s=READY pgs=8842 cs=0 l=1 rev1=1 rx=0x7ff99405efc0 tx=0x7ff994029450).stop 2021-03-09T10:47:56.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:56.546 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016f10 unknown :-1 s=CLOSED pgs=8842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:56.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:56.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:56.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:56.547 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006850 con 0x7ff99c0adc20 2021-03-09T10:47:56.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dbc0 con 0x7ff99c0613a0 2021-03-09T10:47:56.548 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff98000ac10 2021-03-09T10:47:56.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:47:56.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:56.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:47:56.549 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:56.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:56.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:56.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:56.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:56.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:56.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026400 con 0x7ff98000ac10 2021-03-09T10:47:56.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002d890 con 0x7ff98000ac10 2021-03-09T10:47:56.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015510 secure :-1 s=READY pgs=8854 cs=0 l=1 rev1=1 rx=0x7ff994009ec0 tx=0x7ff994014ce0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:56.552 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:47:56.542+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30007 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:47:56.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069350 con 0x7ff98000ac10 2021-03-09T10:47:56.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940aa080 con 0x7ff98000ac10 2021-03-09T10:47:56.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.546+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:47:56.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.546+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:56.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.546+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:56.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:47:56.546+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:01.550 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:01.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:01.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015510 secure :-1 s=READY pgs=8854 cs=0 l=1 rev1=1 rx=0x7ff994009ec0 tx=0x7ff994014ce0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:01.551 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015510 secure :-1 s=READY pgs=8854 cs=0 l=1 rev1=1 rx=0x7ff994009ec0 tx=0x7ff994014ce0).stop 2021-03-09T10:48:01.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:01.552 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015510 unknown :-1 s=CLOSED pgs=8854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:01.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:01.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:01.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:01.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980039a00 con 0x7ff99c0adc20 2021-03-09T10:48:01.553 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a280 con 0x7ff99c0613a0 2021-03-09T10:48:01.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dbc0 con 0x7ff980051540 2021-03-09T10:48:01.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:01.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.542+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:01.554 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.546+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:01.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.546+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:01.555 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.546+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:01.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.546+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:01.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.546+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:01.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.546+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:01.556 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.546+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:01.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.546+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026400 con 0x7ff980051540 2021-03-09T10:48:01.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.546+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002d890 con 0x7ff980051540 2021-03-09T10:48:01.557 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.546+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016d20 secure :-1 s=READY pgs=8864 cs=0 l=1 rev1=1 rx=0x7ff99405d250 tx=0x7ff994084e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:01.558 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:01.550+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30043 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:01.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.554+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405d090 con 0x7ff980051540 2021-03-09T10:48:01.560 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.554+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994014ec0 con 0x7ff980051540 2021-03-09T10:48:01.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.554+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:48:01.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.554+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:01.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.554+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:01.563 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:01.554+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:04.931 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:48:04.934 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:48:06.562 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:06.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:06.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016d20 secure :-1 s=READY pgs=8864 cs=0 l=1 rev1=1 rx=0x7ff99405d250 tx=0x7ff994084e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:06.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016d20 secure :-1 s=READY pgs=8864 cs=0 l=1 rev1=1 rx=0x7ff99405d250 tx=0x7ff994084e60).stop 2021-03-09T10:48:06.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:06.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016d20 unknown :-1 s=CLOSED pgs=8864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:06.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:06.574 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:06.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:06.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980036190 con 0x7ff99c0adc20 2021-03-09T10:48:06.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dbc0 con 0x7ff99c0613a0 2021-03-09T10:48:06.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026400 con 0x7ff98000ac10 2021-03-09T10:48:06.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:06.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:06.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:06.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:06.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.550+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:06.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.554+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:06.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.554+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:06.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.554+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:06.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.554+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:06.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.554+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015970 con 0x7ff98000ac10 2021-03-09T10:48:06.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.554+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002d890 con 0x7ff98000ac10 2021-03-09T10:48:06.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.554+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015480 secure :-1 s=READY pgs=8873 cs=0 l=1 rev1=1 rx=0x7ff99409da90 tx=0x7ff9940852d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:06.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.558+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994014ec0 con 0x7ff98000ac10 2021-03-09T10:48:06.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.558+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069350 con 0x7ff98000ac10 2021-03-09T10:48:06.585 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:06.554+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30082 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:06.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.562+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:48:06.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.562+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:06.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.562+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:06.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:06.562+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:11.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:11.564 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:11.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015480 secure :-1 s=READY pgs=8873 cs=0 l=1 rev1=1 rx=0x7ff99409da90 tx=0x7ff9940852d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:11.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015480 secure :-1 s=READY pgs=8873 cs=0 l=1 rev1=1 rx=0x7ff99409da90 tx=0x7ff9940852d0).stop 2021-03-09T10:48:11.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980015480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:11.565 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980015480 unknown :-1 s=CLOSED pgs=8873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:11.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:11.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:11.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:11.566 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026400 con 0x7ff99c0adc20 2021-03-09T10:48:11.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055350 con 0x7ff99c0613a0 2021-03-09T10:48:11.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020960 con 0x7ff980051540 2021-03-09T10:48:11.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:11.567 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:11.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:11.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:11.568 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:11.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:11.569 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:11.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:11.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:11.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017020 con 0x7ff980051540 2021-03-09T10:48:11.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015970 con 0x7ff980051540 2021-03-09T10:48:11.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.558+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016b30 secure :-1 s=READY pgs=8885 cs=0 l=1 rev1=1 rx=0x7ff99402a430 tx=0x7ff994064090).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:11.571 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:11.562+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30118 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:11.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.562+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409d6f0 con 0x7ff980051540 2021-03-09T10:48:11.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.562+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994069350 con 0x7ff980051540 2021-03-09T10:48:11.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.566+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001da0 con 0x7ff980051540 2021-03-09T10:48:11.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.566+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:11.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.570+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:11.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:11.570+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:16.570 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:16.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:16.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016b30 secure :-1 s=READY pgs=8885 cs=0 l=1 rev1=1 rx=0x7ff99402a430 tx=0x7ff994064090).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:16.571 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016b30 secure :-1 s=READY pgs=8885 cs=0 l=1 rev1=1 rx=0x7ff99402a430 tx=0x7ff994064090).stop 2021-03-09T10:48:16.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:16.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016b30 unknown :-1 s=CLOSED pgs=8885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:16.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:16.572 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:16.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:16.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026400 con 0x7ff99c0adc20 2021-03-09T10:48:16.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055500 con 0x7ff99c0613a0 2021-03-09T10:48:16.573 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800153f0 con 0x7ff98000ac10 2021-03-09T10:48:16.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:16.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:16.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:16.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:16.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:16.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:16.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:16.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:16.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:16.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015530 con 0x7ff98000ac10 2021-03-09T10:48:16.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015c90 con 0x7ff98000ac10 2021-03-09T10:48:16.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.566+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055010 secure :-1 s=READY pgs=8895 cs=0 l=1 rev1=1 rx=0x7ff98c087430 tx=0x7ff98c022f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:16.578 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:16.566+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30157 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:16.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.570+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0450d0 con 0x7ff98000ac10 2021-03-09T10:48:16.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.570+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff98000ac10 2021-03-09T10:48:16.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.574+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff98000ac10 2021-03-09T10:48:16.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.574+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:16.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.574+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:16.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:16.574+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:21.575 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:21.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:21.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055010 secure :-1 s=READY pgs=8895 cs=0 l=1 rev1=1 rx=0x7ff98c087430 tx=0x7ff98c022f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:21.576 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055010 secure :-1 s=READY pgs=8895 cs=0 l=1 rev1=1 rx=0x7ff98c087430 tx=0x7ff98c022f40).stop 2021-03-09T10:48:21.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980055010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:21.577 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980055010 unknown :-1 s=CLOSED pgs=8895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:21.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:21.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:21.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:21.578 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055500 con 0x7ff99c0adc20 2021-03-09T10:48:21.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002de30 con 0x7ff99c0613a0 2021-03-09T10:48:21.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052dd0 con 0x7ff980051540 2021-03-09T10:48:21.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:21.579 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:21.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:21.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:21.580 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:21.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:21.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:21.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:21.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:21.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980016e30 con 0x7ff980051540 2021-03-09T10:48:21.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.570+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017020 con 0x7ff980051540 2021-03-09T10:48:21.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.574+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016940 secure :-1 s=READY pgs=8907 cs=0 l=1 rev1=1 rx=0x7ff99401e7f0 tx=0x7ff99402a460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:21.583 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:21.574+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30193 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:21.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.574+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994068310 con 0x7ff980051540 2021-03-09T10:48:21.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.578+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994068130 con 0x7ff980051540 2021-03-09T10:48:21.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.578+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:48:21.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.578+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:21.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.582+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:21.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:21.582+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:26.581 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:26.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:26.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016940 secure :-1 s=READY pgs=8907 cs=0 l=1 rev1=1 rx=0x7ff99401e7f0 tx=0x7ff99402a460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:26.582 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016940 secure :-1 s=READY pgs=8907 cs=0 l=1 rev1=1 rx=0x7ff99401e7f0 tx=0x7ff99402a460).stop 2021-03-09T10:48:26.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:26.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016940 unknown :-1 s=CLOSED pgs=8907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:26.583 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:26.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:26.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:26.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800153f0 con 0x7ff99c0adc20 2021-03-09T10:48:26.584 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026400 con 0x7ff99c0613a0 2021-03-09T10:48:26.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980052da0 con 0x7ff98000ac10 2021-03-09T10:48:26.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:26.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:26.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:26.585 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:26.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:26.586 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:26.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:26.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:26.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:26.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002dde0 con 0x7ff98000ac10 2021-03-09T10:48:26.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.578+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980016e30 con 0x7ff98000ac10 2021-03-09T10:48:26.589 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:26.578+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30232 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:26.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.582+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 secure :-1 s=READY pgs=8922 cs=0 l=1 rev1=1 rx=0x7ff9940aa600 tx=0x7ff9940aa670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:26.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.582+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940aa080 con 0x7ff98000ac10 2021-03-09T10:48:26.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.582+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994068130 con 0x7ff98000ac10 2021-03-09T10:48:26.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.586+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:48:26.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.586+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:26.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.586+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:26.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:26.586+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:31.587 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:31.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:31.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 secure :-1 s=READY pgs=8922 cs=0 l=1 rev1=1 rx=0x7ff9940aa600 tx=0x7ff9940aa670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:31.588 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 secure :-1 s=READY pgs=8922 cs=0 l=1 rev1=1 rx=0x7ff9940aa600 tx=0x7ff9940aa670).stop 2021-03-09T10:48:31.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980054e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:31.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980054e40 unknown :-1 s=CLOSED pgs=8922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:31.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:31.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:31.589 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:31.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800153f0 con 0x7ff99c0adc20 2021-03-09T10:48:31.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800554d0 con 0x7ff99c0613a0 2021-03-09T10:48:31.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800208c0 con 0x7ff980051540 2021-03-09T10:48:31.590 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:31.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:31.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:31.591 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:31.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:31.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:31.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:31.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:31.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:31.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980016c40 con 0x7ff980051540 2021-03-09T10:48:31.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.582+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002dde0 con 0x7ff980051540 2021-03-09T10:48:31.594 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:31.586+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30268 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:31.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.586+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016750 secure :-1 s=READY pgs=8932 cs=0 l=1 rev1=1 rx=0x7ff994098510 tx=0x7ff994081d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:31.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.586+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994098350 con 0x7ff980051540 2021-03-09T10:48:31.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.590+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098120 con 0x7ff980051540 2021-03-09T10:48:31.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.594+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:48:31.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.594+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:31.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.594+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:31.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:31.594+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:35.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:48:35.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:48:36.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:36.592 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:36.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016750 secure :-1 s=READY pgs=8932 cs=0 l=1 rev1=1 rx=0x7ff994098510 tx=0x7ff994081d40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:36.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016750 secure :-1 s=READY pgs=8932 cs=0 l=1 rev1=1 rx=0x7ff994098510 tx=0x7ff994081d40).stop 2021-03-09T10:48:36.593 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980016750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:36.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980016750 unknown :-1 s=CLOSED pgs=8932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:36.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:36.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800197f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:36.594 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:36.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800153f0 con 0x7ff99c0adc20 2021-03-09T10:48:36.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056e10 con 0x7ff99c0613a0 2021-03-09T10:48:36.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800929c0 con 0x7ff98000ac10 2021-03-09T10:48:36.595 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:36.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:36.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:36.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:36.596 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.586+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800197f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:36.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.590+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:36.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.590+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:36.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.590+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:36.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.590+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:36.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.590+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980092b30 con 0x7ff98000ac10 2021-03-09T10:48:36.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.590+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015530 con 0x7ff98000ac10 2021-03-09T10:48:36.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.590+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800197f0 secure :-1 s=READY pgs=8937 cs=0 l=1 rev1=1 rx=0x7ff98c0a2fd0 tx=0x7ff98c079760).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:36.599 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:36.590+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30307 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:36.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.590+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:48:36.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.590+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04bcc0 con 0x7ff98000ac10 2021-03-09T10:48:36.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.594+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:48:36.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.594+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:36.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.594+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:36.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:36.594+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:41.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800197f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:41.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800197f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:41.597 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800197f0 secure :-1 s=READY pgs=8937 cs=0 l=1 rev1=1 rx=0x7ff98c0a2fd0 tx=0x7ff98c079760).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:41.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800197f0 secure :-1 s=READY pgs=8937 cs=0 l=1 rev1=1 rx=0x7ff98c0a2fd0 tx=0x7ff98c079760).stop 2021-03-09T10:48:41.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800197f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:41.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800197f0 unknown :-1 s=CLOSED pgs=8937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:41.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:41.598 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:41.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:41.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056e10 con 0x7ff99c0adc20 2021-03-09T10:48:41.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014480 con 0x7ff99c0613a0 2021-03-09T10:48:41.599 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:41.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092cc0 con 0x7ff980051540 2021-03-09T10:48:41.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:41.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:41.600 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:41.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:41.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:41.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:41.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:41.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:41.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800208c0 con 0x7ff980051540 2021-03-09T10:48:41.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092b30 con 0x7ff980051540 2021-03-09T10:48:41.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.593+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 secure :-1 s=READY pgs=8949 cs=0 l=1 rev1=1 rx=0x7ff98c08ff70 tx=0x7ff98c00bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:41.603 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:41.593+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30342 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:41.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:48:41.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c072ce0 con 0x7ff980051540 2021-03-09T10:48:41.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00b120 con 0x7ff980051540 2021-03-09T10:48:41.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.597+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:41.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.601+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:41.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:41.601+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:46.601 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:46.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:46.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 secure :-1 s=READY pgs=8949 cs=0 l=1 rev1=1 rx=0x7ff98c08ff70 tx=0x7ff98c00bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:46.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 secure :-1 s=READY pgs=8949 cs=0 l=1 rev1=1 rx=0x7ff98c08ff70 tx=0x7ff98c00bfd0).stop 2021-03-09T10:48:46.602 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980054e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:46.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 unknown :-1 s=CLOSED pgs=8949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:46.603 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:46.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:46.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:46.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056e10 con 0x7ff99c0adc20 2021-03-09T10:48:46.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047830 con 0x7ff99c0613a0 2021-03-09T10:48:46.604 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092e60 con 0x7ff98000ac10 2021-03-09T10:48:46.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:46.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:46.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:46.605 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:46.606 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:46.606 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:46.597+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30381 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:46.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:46.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:46.607 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:46.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:46.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980014480 con 0x7ff98000ac10 2021-03-09T10:48:46.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.597+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980016c40 con 0x7ff98000ac10 2021-03-09T10:48:46.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019750 secure :-1 s=READY pgs=8965 cs=0 l=1 rev1=1 rx=0x7ff994039e10 tx=0x7ff994009e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:46.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994064ed0 con 0x7ff98000ac10 2021-03-09T10:48:46.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.605+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994020220 con 0x7ff98000ac10 2021-03-09T10:48:46.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.609+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:48:46.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.609+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:46.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.609+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:46.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:46.609+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:51.608 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:51.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:51.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019750 secure :-1 s=READY pgs=8965 cs=0 l=1 rev1=1 rx=0x7ff994039e10 tx=0x7ff994009e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:51.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019750 secure :-1 s=READY pgs=8965 cs=0 l=1 rev1=1 rx=0x7ff994039e10 tx=0x7ff994009e60).stop 2021-03-09T10:48:51.609 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:51.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019750 unknown :-1 s=CLOSED pgs=8965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:51.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:51.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:51.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:51.610 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056e10 con 0x7ff99c0adc20 2021-03-09T10:48:51.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047830 con 0x7ff99c0613a0 2021-03-09T10:48:51.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800929c0 con 0x7ff980051540 2021-03-09T10:48:51.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:51.611 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:51.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:51.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:51.612 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:51.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:51.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:51.613 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:51.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:51.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980016de0 con 0x7ff980051540 2021-03-09T10:48:51.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980014480 con 0x7ff980051540 2021-03-09T10:48:51.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.601+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 secure :-1 s=READY pgs=8969 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff994020220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:51.615 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:51.605+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30418 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:51.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.605+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994014120 con 0x7ff980051540 2021-03-09T10:48:51.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.605+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:48:51.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.609+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff980051540 2021-03-09T10:48:51.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.609+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:51.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.609+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:51.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:51.609+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:48:56.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:56.614 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:56.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 secure :-1 s=READY pgs=8969 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff994020220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:56.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 secure :-1 s=READY pgs=8969 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff994020220).stop 2021-03-09T10:48:56.615 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980054e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:56.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 unknown :-1 s=CLOSED pgs=8969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:56.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:56.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:56.616 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:56.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056e10 con 0x7ff99c0adc20 2021-03-09T10:48:56.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047830 con 0x7ff99c0613a0 2021-03-09T10:48:56.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800570b0 con 0x7ff98000ac10 2021-03-09T10:48:56.617 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:48:56.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:56.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:48:56.618 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:56.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:56.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:56.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:56.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:56.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:56.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800929a0 con 0x7ff98000ac10 2021-03-09T10:48:56.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980016de0 con 0x7ff98000ac10 2021-03-09T10:48:56.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.609+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019410 secure :-1 s=READY pgs=8984 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994064ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:56.621 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:48:56.609+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30456 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:48:56.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.613+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994014120 con 0x7ff98000ac10 2021-03-09T10:48:56.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.617+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994014120 con 0x7ff98000ac10 2021-03-09T10:48:56.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.617+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff98000ac10 2021-03-09T10:48:56.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.617+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:56.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.621+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:56.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:48:56.621+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:01.619 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:49:01.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:01.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019410 secure :-1 s=READY pgs=8984 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994064ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:01.620 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019410 secure :-1 s=READY pgs=8984 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994064ed0).stop 2021-03-09T10:49:01.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:01.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019410 unknown :-1 s=CLOSED pgs=8984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:01.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:01.621 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:01.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:01.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800570b0 con 0x7ff99c0adc20 2021-03-09T10:49:01.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016f80 con 0x7ff99c0613a0 2021-03-09T10:49:01.622 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800208c0 con 0x7ff980051540 2021-03-09T10:49:01.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:01.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:01.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:01.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:01.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:01.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:01.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:01.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:01.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:01.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980055330 con 0x7ff980051540 2021-03-09T10:49:01.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980092b30 con 0x7ff980051540 2021-03-09T10:49:01.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.613+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 secure :-1 s=READY pgs=8992 cs=0 l=1 rev1=1 rx=0x7ff98c0172e0 tx=0x7ff98c0873a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:01.627 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:01.617+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30492 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:01.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.617+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff980051540 2021-03-09T10:49:01.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.617+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04c140 con 0x7ff980051540 2021-03-09T10:49:01.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.621+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079b80 con 0x7ff980051540 2021-03-09T10:49:01.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.621+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:01.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.621+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:01.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:01.621+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:05.274 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:49:05.277 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:49:06.624 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:06.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:06.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 secure :-1 s=READY pgs=8992 cs=0 l=1 rev1=1 rx=0x7ff98c0172e0 tx=0x7ff98c0873a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:06.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 secure :-1 s=READY pgs=8992 cs=0 l=1 rev1=1 rx=0x7ff98c0172e0 tx=0x7ff98c0873a0).stop 2021-03-09T10:49:06.625 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980054e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:06.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980054e40 unknown :-1 s=CLOSED pgs=8992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:06.626 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:06.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:06.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:06.627 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:06.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:06.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:06.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:06.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800570b0 con 0x7ff99c0adc20 2021-03-09T10:49:06.628 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092cd0 con 0x7ff99c0613a0 2021-03-09T10:49:06.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019860 con 0x7ff98000ac10 2021-03-09T10:49:06.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:06.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:06.629 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:06.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:06.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.621+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:06.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.625+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800199a0 con 0x7ff98000ac10 2021-03-09T10:49:06.630 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.625+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800929a0 con 0x7ff98000ac10 2021-03-09T10:49:06.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.625+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019370 secure :-1 s=READY pgs=9001 cs=0 l=1 rev1=1 rx=0x7ff98008d460 tx=0x7ff99408d230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:06.631 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:06.625+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30531 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:06.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.629+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99406f2a0 con 0x7ff98000ac10 2021-03-09T10:49:06.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.629+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99402a180 con 0x7ff98000ac10 2021-03-09T10:49:06.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.633+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff98000ac10 2021-03-09T10:49:06.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.633+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:06.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.633+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:06.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:06.633+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:11.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:11.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:11.631 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019370 secure :-1 s=READY pgs=9001 cs=0 l=1 rev1=1 rx=0x7ff98008d460 tx=0x7ff99408d230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:11.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019370 secure :-1 s=READY pgs=9001 cs=0 l=1 rev1=1 rx=0x7ff98008d460 tx=0x7ff99408d230).stop 2021-03-09T10:49:11.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980019370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:11.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980019370 unknown :-1 s=CLOSED pgs=9001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:11.632 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:11.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:11.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:11.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019860 con 0x7ff99c0adc20 2021-03-09T10:49:11.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046740 con 0x7ff99c0613a0 2021-03-09T10:49:11.633 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800570b0 con 0x7ff980051540 2021-03-09T10:49:11.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:11.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:11.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:11.634 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:11.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.625+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:11.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.629+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:11.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.629+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:11.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.629+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:11.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.629+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:11.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.629+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800208c0 con 0x7ff980051540 2021-03-09T10:49:11.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.629+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055330 con 0x7ff980051540 2021-03-09T10:49:11.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.629+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092800 secure :-1 s=READY pgs=9014 cs=0 l=1 rev1=1 rx=0x7ff980018120 tx=0x7ff98c0871d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:11.638 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:11.629+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30568 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:11.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.633+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c07de90 con 0x7ff980051540 2021-03-09T10:49:11.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.637+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07de90 con 0x7ff980051540 2021-03-09T10:49:11.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.641+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:49:11.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.641+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:11.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.641+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:11.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:11.641+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:16.635 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:16.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:16.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092800 secure :-1 s=READY pgs=9014 cs=0 l=1 rev1=1 rx=0x7ff980018120 tx=0x7ff98c0871d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:16.636 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092800 secure :-1 s=READY pgs=9014 cs=0 l=1 rev1=1 rx=0x7ff980018120 tx=0x7ff98c0871d0).stop 2021-03-09T10:49:16.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:16.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092800 unknown :-1 s=CLOSED pgs=9014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:16.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:16.637 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:16.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:16.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800468e0 con 0x7ff99c0adc20 2021-03-09T10:49:16.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092cf0 con 0x7ff99c0613a0 2021-03-09T10:49:16.638 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800208c0 con 0x7ff98000ac10 2021-03-09T10:49:16.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:16.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:16.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:16.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:16.639 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:16.640 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:16.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:16.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:16.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:16.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980057060 con 0x7ff98000ac10 2021-03-09T10:49:16.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.633+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800199a0 con 0x7ff98000ac10 2021-03-09T10:49:16.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.637+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 secure :-1 s=READY pgs=9023 cs=0 l=1 rev1=1 rx=0x7ff99408d310 tx=0x7ff994081d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:16.642 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:16.637+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30606 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:16.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.637+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99402a180 con 0x7ff98000ac10 2021-03-09T10:49:16.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.641+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940343a0 con 0x7ff98000ac10 2021-03-09T10:49:16.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.645+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff98000ac10 2021-03-09T10:49:16.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.645+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:16.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.645+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:16.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:16.645+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:21.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:21.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:21.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 secure :-1 s=READY pgs=9023 cs=0 l=1 rev1=1 rx=0x7ff99408d310 tx=0x7ff994081d40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:21.641 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 secure :-1 s=READY pgs=9023 cs=0 l=1 rev1=1 rx=0x7ff99408d310 tx=0x7ff994081d40).stop 2021-03-09T10:49:21.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800968b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:21.642 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800968b0 unknown :-1 s=CLOSED pgs=9023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:21.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800927d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:21.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:21.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:21.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094260 con 0x7ff99c0adc20 2021-03-09T10:49:21.643 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d800 con 0x7ff99c0613a0 2021-03-09T10:49:21.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022a40 con 0x7ff980051540 2021-03-09T10:49:21.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:21.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800927d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:21.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:21.644 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:21.645 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.637+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:21.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.641+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:21.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.641+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:21.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.641+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:21.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.641+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:21.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.641+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800208c0 con 0x7ff980051540 2021-03-09T10:49:21.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.641+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980057060 con 0x7ff980051540 2021-03-09T10:49:21.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.641+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800927d0 secure :-1 s=READY pgs=9033 cs=0 l=1 rev1=1 rx=0x7ff994009ec0 tx=0x7ff99408ef80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:21.647 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:21.641+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30643 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:21.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.641+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994069f70 con 0x7ff980051540 2021-03-09T10:49:21.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.641+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:49:21.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.645+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994070070 con 0x7ff980051540 2021-03-09T10:49:21.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.645+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:21.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.649+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:21.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:21.649+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:26.646 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800927d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:26.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800927d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:26.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800927d0 secure :-1 s=READY pgs=9033 cs=0 l=1 rev1=1 rx=0x7ff994009ec0 tx=0x7ff99408ef80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:26.647 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800927d0 secure :-1 s=READY pgs=9033 cs=0 l=1 rev1=1 rx=0x7ff994009ec0 tx=0x7ff99408ef80).stop 2021-03-09T10:49:26.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800927d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:26.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800927d0 unknown :-1 s=CLOSED pgs=9033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:26.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:26.648 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:26.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:26.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d030 con 0x7ff99c0adc20 2021-03-09T10:49:26.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092cc0 con 0x7ff99c0613a0 2021-03-09T10:49:26.649 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980089430 con 0x7ff98000ac10 2021-03-09T10:49:26.650 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:26.650 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:26.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:26.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:26.651 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.641+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:26.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.645+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:26.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.645+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:26.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.645+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:26.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.645+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:26.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.645+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002d6b0 con 0x7ff98000ac10 2021-03-09T10:49:26.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.645+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800208c0 con 0x7ff98000ac10 2021-03-09T10:49:26.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.645+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9046 cs=0 l=1 rev1=1 rx=0x7ff994068ea0 tx=0x7ff994002ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:26.653 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:26.645+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30681 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:26.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.657+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994006b30 con 0x7ff98000ac10 2021-03-09T10:49:26.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.657+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940853c0 con 0x7ff98000ac10 2021-03-09T10:49:26.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.657+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff98000ac10 2021-03-09T10:49:26.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.657+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:26.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.657+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:26.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:26.657+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:31.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:31.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:31.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9046 cs=0 l=1 rev1=1 rx=0x7ff994068ea0 tx=0x7ff994002ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:31.652 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9046 cs=0 l=1 rev1=1 rx=0x7ff994068ea0 tx=0x7ff994002ed0).stop 2021-03-09T10:49:31.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:31.653 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=CLOSED pgs=9046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:31.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:31.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:31.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:31.654 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800208c0 con 0x7ff99c0adc20 2021-03-09T10:49:31.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092b20 con 0x7ff99c0613a0 2021-03-09T10:49:31.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092c60 con 0x7ff980051540 2021-03-09T10:49:31.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:31.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:31.655 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:31.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:31.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:31.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:31.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:31.656 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:31.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:31.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022850 con 0x7ff980051540 2021-03-09T10:49:31.657 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980055330 con 0x7ff980051540 2021-03-09T10:49:31.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.649+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 secure :-1 s=READY pgs=9055 cs=0 l=1 rev1=1 rx=0x7ff98001e780 tx=0x7ff98c003e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:31.658 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:31.653+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30718 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:31.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.653+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980051540 2021-03-09T10:49:31.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.653+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980051540 2021-03-09T10:49:31.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.657+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c07e150 con 0x7ff980051540 2021-03-09T10:49:31.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.657+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:31.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.661+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:31.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:31.661+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:35.320 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:49:35.323 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:49:36.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:36.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:36.658 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 secure :-1 s=READY pgs=9055 cs=0 l=1 rev1=1 rx=0x7ff98001e780 tx=0x7ff98c003e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:36.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 secure :-1 s=READY pgs=9055 cs=0 l=1 rev1=1 rx=0x7ff98001e780 tx=0x7ff98c003e80).stop 2021-03-09T10:49:36.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:36.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092630 unknown :-1 s=CLOSED pgs=9055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:36.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:36.659 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:36.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:36.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800229e0 con 0x7ff99c0adc20 2021-03-09T10:49:36.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022b20 con 0x7ff99c0613a0 2021-03-09T10:49:36.660 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014410 con 0x7ff98000ac10 2021-03-09T10:49:36.661 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:36.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:36.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:36.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:36.662 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.653+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:36.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.657+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:36.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.657+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:36.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.657+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:36.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.657+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:36.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.657+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980014550 con 0x7ff98000ac10 2021-03-09T10:49:36.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.657+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002d6b0 con 0x7ff98000ac10 2021-03-09T10:49:36.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.657+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9066 cs=0 l=1 rev1=1 rx=0x7ff994029510 tx=0x7ff99401cac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:36.665 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:36.657+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30756 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:36.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.661+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994014e80 con 0x7ff98000ac10 2021-03-09T10:49:36.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.661+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff98000ac10 2021-03-09T10:49:36.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.669+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:49:36.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.669+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:36.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.669+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:36.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:36.669+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:41.663 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:41.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:41.664 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9066 cs=0 l=1 rev1=1 rx=0x7ff994029510 tx=0x7ff99401cac0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:41.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9066 cs=0 l=1 rev1=1 rx=0x7ff994029510 tx=0x7ff99401cac0).stop 2021-03-09T10:49:41.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:41.665 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=CLOSED pgs=9066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:41.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:41.666 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:41.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:41.667 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014410 con 0x7ff99c0adc20 2021-03-09T10:49:41.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020870 con 0x7ff99c0613a0 2021-03-09T10:49:41.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092a80 con 0x7ff980051540 2021-03-09T10:49:41.668 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:41.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:41.669 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:41.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:41.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:41.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:41.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:41.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:41.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:41.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff980051540 2021-03-09T10:49:41.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.661+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022850 con 0x7ff980051540 2021-03-09T10:49:41.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.665+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092590 secure :-1 s=READY pgs=9078 cs=0 l=1 rev1=1 rx=0x7ff980004440 tx=0x7ff98c00bef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:41.673 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:41.665+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30793 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:41.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.665+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c084e80 con 0x7ff980051540 2021-03-09T10:49:41.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.669+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c084e80 con 0x7ff980051540 2021-03-09T10:49:41.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.673+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a99d0 con 0x7ff980051540 2021-03-09T10:49:41.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.673+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:41.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.673+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:41.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:41.673+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:46.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:46.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:46.670 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092590 secure :-1 s=READY pgs=9078 cs=0 l=1 rev1=1 rx=0x7ff980004440 tx=0x7ff98c00bef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:46.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092590 secure :-1 s=READY pgs=9078 cs=0 l=1 rev1=1 rx=0x7ff980004440 tx=0x7ff98c00bef0).stop 2021-03-09T10:49:46.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980092590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:46.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980092590 unknown :-1 s=CLOSED pgs=9078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:46.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:46.671 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:46.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:46.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014410 con 0x7ff99c0adc20 2021-03-09T10:49:46.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c420 con 0x7ff99c0613a0 2021-03-09T10:49:46.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092a80 con 0x7ff98000ac10 2021-03-09T10:49:46.672 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:46.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:46.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:46.673 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:46.674 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.665+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:46.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.669+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:46.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.669+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:46.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.669+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:46.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.669+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:46.675 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.669+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096a40 con 0x7ff98000ac10 2021-03-09T10:49:46.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.669+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:49:46.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.669+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9091 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c04bfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:46.676 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:46.669+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30831 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:46.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.673+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c022cf0 con 0x7ff98000ac10 2021-03-09T10:49:46.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.673+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c022cf0 con 0x7ff98000ac10 2021-03-09T10:49:46.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00b330 con 0x7ff98000ac10 2021-03-09T10:49:46.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.677+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:46.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.677+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:46.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:46.677+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:51.676 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.669+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:51.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.669+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:51.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.669+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9091 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c04bfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:51.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.669+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9091 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c04bfc0).stop 2021-03-09T10:49:51.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.669+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:51.677 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.669+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=CLOSED pgs=9091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:51.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:51.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:51.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:51.678 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014410 con 0x7ff99c0adc20 2021-03-09T10:49:51.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020ba0 con 0x7ff99c0613a0 2021-03-09T10:49:51.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003b1c0 con 0x7ff980051540 2021-03-09T10:49:51.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:51.679 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:51.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:51.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:51.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:51.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:51.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:51.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:51.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:51.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff980051540 2021-03-09T10:49:51.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980014550 con 0x7ff980051540 2021-03-09T10:49:51.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.673+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 secure :-1 s=READY pgs=9097 cs=0 l=1 rev1=1 rx=0x7ff9940292e0 tx=0x7ff994081dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:51.683 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:51.677+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30868 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:51.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d1c0 con 0x7ff980051540 2021-03-09T10:49:51.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940ae3a0 con 0x7ff980051540 2021-03-09T10:49:51.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.681+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:49:51.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.681+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:51.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.681+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:51.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:51.681+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:49:56.680 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980096bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:56.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980096bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:56.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 secure :-1 s=READY pgs=9097 cs=0 l=1 rev1=1 rx=0x7ff9940292e0 tx=0x7ff994081dc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:56.681 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 secure :-1 s=READY pgs=9097 cs=0 l=1 rev1=1 rx=0x7ff9940292e0 tx=0x7ff994081dc0).stop 2021-03-09T10:49:56.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980096bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:56.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 unknown :-1 s=CLOSED pgs=9097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:56.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:56.682 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:56.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:56.683 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800285e0 con 0x7ff99c0adc20 2021-03-09T10:49:56.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047890 con 0x7ff99c0613a0 2021-03-09T10:49:56.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020ba0 con 0x7ff98000ac10 2021-03-09T10:49:56.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:49:56.684 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:49:56.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:56.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:56.685 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.677+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:56.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:56.686 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.681+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:56.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:56.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.681+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:56.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000c420 con 0x7ff98000ac10 2021-03-09T10:49:56.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.681+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:49:56.688 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:49:56.681+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30906 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:49:56.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.681+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9112 cs=0 l=1 rev1=1 rx=0x7ff994001300 tx=0x7ff99400cdf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:56.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.685+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98000ac10 2021-03-09T10:49:56.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.689+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98000ac10 2021-03-09T10:49:56.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.693+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff98000ac10 2021-03-09T10:49:56.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.693+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:56.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.693+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:56.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:49:56.693+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:01.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:01.687 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:01.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9112 cs=0 l=1 rev1=1 rx=0x7ff994001300 tx=0x7ff99400cdf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:01.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9112 cs=0 l=1 rev1=1 rx=0x7ff994001300 tx=0x7ff99400cdf0).stop 2021-03-09T10:50:01.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:01.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=CLOSED pgs=9112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:01.688 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:01.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:01.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:01.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020ba0 con 0x7ff99c0adc20 2021-03-09T10:50:01.689 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022d20 con 0x7ff99c0613a0 2021-03-09T10:50:01.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dac0 con 0x7ff980051540 2021-03-09T10:50:01.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:01.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:01.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:01.690 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.680+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:01.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.684+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:01.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.684+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:01.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.684+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:01.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.684+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:01.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.684+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:01.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.684+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff980051540 2021-03-09T10:50:01.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.684+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096a40 con 0x7ff980051540 2021-03-09T10:50:01.693 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:01.684+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30943 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:01.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.684+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 secure :-1 s=READY pgs=9120 cs=0 l=1 rev1=1 rx=0x7ff98000e5c0 tx=0x7ff98c0066c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:01.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.688+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c052130 con 0x7ff980051540 2021-03-09T10:50:01.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.688+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c098d40 con 0x7ff980051540 2021-03-09T10:50:01.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.692+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980051540 2021-03-09T10:50:01.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.692+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:01.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.692+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:01.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:01.692+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:05.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:50:05.420 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:50:06.691 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980096bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:06.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980096bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:06.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 secure :-1 s=READY pgs=9120 cs=0 l=1 rev1=1 rx=0x7ff98000e5c0 tx=0x7ff98c0066c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:06.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 secure :-1 s=READY pgs=9120 cs=0 l=1 rev1=1 rx=0x7ff98000e5c0 tx=0x7ff98c0066c0).stop 2021-03-09T10:50:06.692 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980096bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:06.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980096bd0 unknown :-1 s=CLOSED pgs=9120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:06.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:06.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:06.693 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:06.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027980 con 0x7ff99c0adc20 2021-03-09T10:50:06.694 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800554f0 con 0x7ff99c0613a0 2021-03-09T10:50:06.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980096a40 con 0x7ff98000ac10 2021-03-09T10:50:06.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:06.695 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:06.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:06.696 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:06.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:06.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:06.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:06.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:06.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:06.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800285e0 con 0x7ff98000ac10 2021-03-09T10:50:06.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.688+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:50:06.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.692+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9129 cs=0 l=1 rev1=1 rx=0x7ff980012ee0 tx=0x7ff98c07ee30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:06.702 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:06.692+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30981 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:06.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.692+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c098d40 con 0x7ff98000ac10 2021-03-09T10:50:06.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.692+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff98000ac10 2021-03-09T10:50:06.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.696+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c09a1e0 con 0x7ff98000ac10 2021-03-09T10:50:06.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.696+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:06.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.696+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:06.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:06.696+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:11.697 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:11.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:11.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9129 cs=0 l=1 rev1=1 rx=0x7ff980012ee0 tx=0x7ff98c07ee30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:11.698 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 secure :-1 s=READY pgs=9129 cs=0 l=1 rev1=1 rx=0x7ff980012ee0 tx=0x7ff98c07ee30).stop 2021-03-09T10:50:11.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980014980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:11.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980014980 unknown :-1 s=CLOSED pgs=9129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:11.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:11.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:11.699 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:11.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027980 con 0x7ff99c0adc20 2021-03-09T10:50:11.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff99c0613a0 2021-03-09T10:50:11.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff980051540 2021-03-09T10:50:11.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:11.700 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:11.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:11.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:11.701 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:11.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:11.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:11.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:11.702 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:11.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022b80 con 0x7ff980051540 2021-03-09T10:50:11.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.692+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800285e0 con 0x7ff980051540 2021-03-09T10:50:11.703 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.696+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800476f0 secure :-1 s=READY pgs=9141 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c079700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:11.704 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:11.696+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31018 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:11.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.696+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079b60 con 0x7ff980051540 2021-03-09T10:50:11.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.700+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08f2f0 con 0x7ff980051540 2021-03-09T10:50:11.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.704+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079b60 con 0x7ff980051540 2021-03-09T10:50:11.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.704+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:11.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.704+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:11.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:11.704+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:16.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800476f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:16.704 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800476f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:16.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800476f0 secure :-1 s=READY pgs=9141 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c079700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:16.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800476f0 secure :-1 s=READY pgs=9141 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c079700).stop 2021-03-09T10:50:16.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800476f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:16.705 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800476f0 unknown :-1 s=CLOSED pgs=9141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:16.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:16.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:16.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:16.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027980 con 0x7ff99c0adc20 2021-03-09T10:50:16.706 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0613a0 2021-03-09T10:50:16.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff98000ac10 2021-03-09T10:50:16.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:16.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:16.707 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:16.708 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:16.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:16.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:16.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:16.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:16.709 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:16.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:50:16.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.700+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000c420 con 0x7ff98000ac10 2021-03-09T10:50:16.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.704+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013800 secure :-1 s=READY pgs=9152 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:16.711 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:16.704+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31056 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:16.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.704+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400ceb0 con 0x7ff98000ac10 2021-03-09T10:50:16.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.712+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400ceb0 con 0x7ff98000ac10 2021-03-09T10:50:16.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.712+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:50:16.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.712+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:16.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.712+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:16.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:16.712+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:21.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:21.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:21.710 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013800 secure :-1 s=READY pgs=9152 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:21.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013800 secure :-1 s=READY pgs=9152 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99400dfc0).stop 2021-03-09T10:50:21.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:21.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013800 unknown :-1 s=CLOSED pgs=9152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:21.711 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:21.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:21.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:21.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027980 con 0x7ff99c0adc20 2021-03-09T10:50:21.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0613a0 2021-03-09T10:50:21.712 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800576d0 con 0x7ff980051540 2021-03-09T10:50:21.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:21.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:21.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:21.713 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:21.714 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.704+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:21.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.708+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:21.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.708+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:21.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.708+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:21.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.708+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:21.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.708+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096a40 con 0x7ff980051540 2021-03-09T10:50:21.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.708+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff980051540 2021-03-09T10:50:21.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.708+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e460 secure :-1 s=READY pgs=9162 cs=0 l=1 rev1=1 rx=0x7ff98000cd80 tx=0x7ff9940689a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:21.716 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:21.708+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31093 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:21.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.712+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:50:21.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.712+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:50:21.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.716+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994094130 con 0x7ff980051540 2021-03-09T10:50:21.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.716+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:21.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.716+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:21.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:21.716+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:26.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.708+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:50:26.715 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.708+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:26.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.708+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e460 secure :-1 s=READY pgs=9162 cs=0 l=1 rev1=1 rx=0x7ff98000cd80 tx=0x7ff9940689a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:26.716 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.708+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e460 secure :-1 s=READY pgs=9162 cs=0 l=1 rev1=1 rx=0x7ff98000cd80 tx=0x7ff9940689a0).stop 2021-03-09T10:50:26.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.708+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:26.717 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.708+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e460 unknown :-1 s=CLOSED pgs=9162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:26.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:26.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:26.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:26.718 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0adc20 2021-03-09T10:50:26.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004bbb0 con 0x7ff99c0613a0 2021-03-09T10:50:26.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027980 con 0x7ff98000ac10 2021-03-09T10:50:26.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:26.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:26.719 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:26.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:26.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:26.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:26.720 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:26.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:26.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:26.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:50:26.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096a40 con 0x7ff98000ac10 2021-03-09T10:50:26.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.712+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 secure :-1 s=READY pgs=9173 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff994031ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:26.722 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:26.716+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31131 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:26.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.716+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940aa080 con 0x7ff98000ac10 2021-03-09T10:50:26.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.716+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400ce60 con 0x7ff98000ac10 2021-03-09T10:50:26.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.720+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff98000ac10 2021-03-09T10:50:26.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.720+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:26.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.720+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:26.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:26.720+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:31.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:31.721 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:31.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 secure :-1 s=READY pgs=9173 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff994031ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:31.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 secure :-1 s=READY pgs=9173 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff994031ef0).stop 2021-03-09T10:50:31.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800137c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:31.722 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 unknown :-1 s=CLOSED pgs=9173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:31.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:31.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:31.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:31.723 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002ffc0 con 0x7ff99c0adc20 2021-03-09T10:50:31.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0613a0 2021-03-09T10:50:31.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027980 con 0x7ff980051540 2021-03-09T10:50:31.724 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:31.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:31.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:31.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:31.725 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:31.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:31.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:31.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:31.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.716+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:31.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.720+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004bbb0 con 0x7ff980051540 2021-03-09T10:50:31.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.720+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff980051540 2021-03-09T10:50:31.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.720+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 secure :-1 s=READY pgs=9183 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99400ce60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:31.728 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:31.720+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31168 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:31.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.720+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940ab510 con 0x7ff980051540 2021-03-09T10:50:31.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.724+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400a630 con 0x7ff980051540 2021-03-09T10:50:31.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.728+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff980051540 2021-03-09T10:50:31.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.728+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:31.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.728+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:31.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:31.728+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:35.463 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:50:35.466 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:50:36.726 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:36.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:36.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 secure :-1 s=READY pgs=9183 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99400ce60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:36.727 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 secure :-1 s=READY pgs=9183 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99400ce60).stop 2021-03-09T10:50:36.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:36.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 unknown :-1 s=CLOSED pgs=9183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:36.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:36.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:36.728 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:36.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0adc20 2021-03-09T10:50:36.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980090180 con 0x7ff99c0613a0 2021-03-09T10:50:36.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027980 con 0x7ff98000ac10 2021-03-09T10:50:36.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:36.729 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:36.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:36.730 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.720+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:36.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.724+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:36.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.724+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:36.731 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.724+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:36.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.724+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:36.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.724+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:36.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.724+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002ff70 con 0x7ff98000ac10 2021-03-09T10:50:36.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.724+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004bbb0 con 0x7ff98000ac10 2021-03-09T10:50:36.733 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:36.724+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31206 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:36.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.728+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 secure :-1 s=READY pgs=9196 cs=0 l=1 rev1=1 rx=0x7ff9940857c0 tx=0x7ff99408ef50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:36.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.728+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940853d0 con 0x7ff98000ac10 2021-03-09T10:50:36.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.732+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940851f0 con 0x7ff98000ac10 2021-03-09T10:50:36.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.736+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff98000ac10 2021-03-09T10:50:36.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.736+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:36.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.736+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:36.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:36.736+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:41.732 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:41.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:41.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 secure :-1 s=READY pgs=9196 cs=0 l=1 rev1=1 rx=0x7ff9940857c0 tx=0x7ff99408ef50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:41.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 secure :-1 s=READY pgs=9196 cs=0 l=1 rev1=1 rx=0x7ff9940857c0 tx=0x7ff99408ef50).stop 2021-03-09T10:50:41.733 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800137c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:41.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800137c0 unknown :-1 s=CLOSED pgs=9196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:41.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:41.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:41.734 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:41.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092f00 con 0x7ff99c0adc20 2021-03-09T10:50:41.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027980 con 0x7ff99c0613a0 2021-03-09T10:50:41.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b710 con 0x7ff980051540 2021-03-09T10:50:41.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:41.735 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:41.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:41.736 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:41.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.728+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:41.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.732+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:41.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.732+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:41.737 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.732+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:41.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.732+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:41.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.732+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020820 con 0x7ff980051540 2021-03-09T10:50:41.738 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.732+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002ff70 con 0x7ff980051540 2021-03-09T10:50:41.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.732+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 secure :-1 s=READY pgs=9210 cs=0 l=1 rev1=1 rx=0x7ff99407a320 tx=0x7ff9940856b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:41.739 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:41.732+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31242 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:41.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.736+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940851f0 con 0x7ff980051540 2021-03-09T10:50:41.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.740+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940851f0 con 0x7ff980051540 2021-03-09T10:50:41.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.740+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994060400 con 0x7ff980051540 2021-03-09T10:50:41.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.740+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:41.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.740+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:41.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:41.740+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:46.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:46.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:46.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 secure :-1 s=READY pgs=9210 cs=0 l=1 rev1=1 rx=0x7ff99407a320 tx=0x7ff9940856b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:46.739 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 secure :-1 s=READY pgs=9210 cs=0 l=1 rev1=1 rx=0x7ff99407a320 tx=0x7ff9940856b0).stop 2021-03-09T10:50:46.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:46.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 unknown :-1 s=CLOSED pgs=9210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:46.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:46.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800135d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:46.740 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:46.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980027980 con 0x7ff99c0adc20 2021-03-09T10:50:46.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff99c0613a0 2021-03-09T10:50:46.741 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff98000ac10 2021-03-09T10:50:46.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.732+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:46.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.736+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:46.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.736+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:46.742 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.736+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:46.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.736+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800135d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:46.743 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.736+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:46.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.736+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:46.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.736+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:46.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.736+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:46.744 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.736+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013ac0 con 0x7ff98000ac10 2021-03-09T10:50:46.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.736+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022b80 con 0x7ff98000ac10 2021-03-09T10:50:46.745 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:46.740+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31281 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:46.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.740+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800135d0 secure :-1 s=READY pgs=9219 cs=0 l=1 rev1=1 rx=0x7ff98c006aa0 tx=0x7ff98c07cdb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:46.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.740+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a9120 con 0x7ff98000ac10 2021-03-09T10:50:46.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a9120 con 0x7ff98000ac10 2021-03-09T10:50:46.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:50:46.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.744+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:46.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.748+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:46.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:46.748+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:51.745 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800135d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:51.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800135d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:51.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800135d0 secure :-1 s=READY pgs=9219 cs=0 l=1 rev1=1 rx=0x7ff98c006aa0 tx=0x7ff98c07cdb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:51.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800135d0 secure :-1 s=READY pgs=9219 cs=0 l=1 rev1=1 rx=0x7ff98c006aa0 tx=0x7ff98c07cdb0).stop 2021-03-09T10:50:51.746 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800135d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:51.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800135d0 unknown :-1 s=CLOSED pgs=9219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:51.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:51.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:51.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:51.747 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98003dfc0 con 0x7ff99c0adc20 2021-03-09T10:50:51.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0613a0 2021-03-09T10:50:51.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013c50 con 0x7ff980051540 2021-03-09T10:50:51.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:51.748 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:51.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:51.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:51.749 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:51.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:51.750 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:51.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:51.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:51.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013dc0 con 0x7ff980051540 2021-03-09T10:50:51.751 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.740+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002ff70 con 0x7ff980051540 2021-03-09T10:50:51.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.744+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 secure :-1 s=READY pgs=9225 cs=0 l=1 rev1=1 rx=0x7ff9940756f0 tx=0x7ff9940753d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:51.752 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:51.744+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31317 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:51.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940368f0 con 0x7ff980051540 2021-03-09T10:50:51.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff980051540 2021-03-09T10:50:51.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:50:51.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.752+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:51.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.752+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:51.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:51.752+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:50:56.752 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:56.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:56.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 secure :-1 s=READY pgs=9225 cs=0 l=1 rev1=1 rx=0x7ff9940756f0 tx=0x7ff9940753d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:56.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 secure :-1 s=READY pgs=9225 cs=0 l=1 rev1=1 rx=0x7ff9940756f0 tx=0x7ff9940753d0).stop 2021-03-09T10:50:56.753 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98008e3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:56.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98008e3c0 unknown :-1 s=CLOSED pgs=9225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:56.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:56.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:56.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:56.754 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013c50 con 0x7ff99c0adc20 2021-03-09T10:50:56.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013f50 con 0x7ff99c0613a0 2021-03-09T10:50:56.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.744+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980014090 con 0x7ff98000ac10 2021-03-09T10:50:56.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:50:56.755 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:50:56.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:56.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:56.756 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:56.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:56.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:56.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:56.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:56.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002fd80 con 0x7ff98000ac10 2021-03-09T10:50:56.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.748+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013dc0 con 0x7ff98000ac10 2021-03-09T10:50:56.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.752+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013560 secure :-1 s=READY pgs=9240 cs=0 l=1 rev1=1 rx=0x7ff99402ad90 tx=0x7ff994075360).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:56.759 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:50:56.752+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31356 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:50:56.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:50:56.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.756+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009d80 con 0x7ff98000ac10 2021-03-09T10:50:56.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:50:56.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.760+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:56.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.760+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:56.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:50:56.760+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:01.757 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:01.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:01.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013560 secure :-1 s=READY pgs=9240 cs=0 l=1 rev1=1 rx=0x7ff99402ad90 tx=0x7ff994075360).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:01.758 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013560 secure :-1 s=READY pgs=9240 cs=0 l=1 rev1=1 rx=0x7ff99402ad90 tx=0x7ff994075360).stop 2021-03-09T10:51:01.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:01.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013560 unknown :-1 s=CLOSED pgs=9240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:01.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:01.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:01.759 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:01.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dde0 con 0x7ff99c0adc20 2021-03-09T10:51:01.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800195a0 con 0x7ff99c0613a0 2021-03-09T10:51:01.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff980051540 2021-03-09T10:51:01.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:01.760 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:01.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:01.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:01.761 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.752+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:01.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.756+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:01.762 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.756+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:01.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.756+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:01.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.756+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:01.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.756+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980014140 con 0x7ff980051540 2021-03-09T10:51:01.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.756+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002fd80 con 0x7ff980051540 2021-03-09T10:51:01.763 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.756+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013c50 secure :-1 s=READY pgs=9250 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994009df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:01.764 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:01.756+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31392 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:01.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b7e0 con 0x7ff980051540 2021-03-09T10:51:01.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.768+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b600 con 0x7ff980051540 2021-03-09T10:51:01.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.768+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:51:01.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.768+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:01.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.768+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:01.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:01.768+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:05.510 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:51:05.519 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:51:06.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:06.764 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:06.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013c50 secure :-1 s=READY pgs=9250 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994009df0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:06.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013c50 secure :-1 s=READY pgs=9250 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994009df0).stop 2021-03-09T10:51:06.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:06.765 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013c50 unknown :-1 s=CLOSED pgs=9250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:06.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:06.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800134c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:06.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:06.766 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dde0 con 0x7ff99c0adc20 2021-03-09T10:51:06.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800199a0 con 0x7ff99c0613a0 2021-03-09T10:51:06.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980019ae0 con 0x7ff98000ac10 2021-03-09T10:51:06.767 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:06.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:06.769 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:06.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:06.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:06.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:06.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:06.770 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:06.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:06.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800477a0 con 0x7ff98000ac10 2021-03-09T10:51:06.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013ac0 con 0x7ff98000ac10 2021-03-09T10:51:06.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.760+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800134c0 secure :-1 s=READY pgs=9257 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c0a2e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:06.772 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:06.764+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31431 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:06.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2b70 con 0x7ff98000ac10 2021-03-09T10:51:06.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2b70 con 0x7ff98000ac10 2021-03-09T10:51:06.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.768+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a33b0 con 0x7ff98000ac10 2021-03-09T10:51:06.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.768+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:06.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.768+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:06.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:06.768+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:11.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800134c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:11.771 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800134c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:11.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800134c0 secure :-1 s=READY pgs=9257 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c0a2e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:11.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800134c0 secure :-1 s=READY pgs=9257 cs=0 l=1 rev1=1 rx=0x7ff98c07a450 tx=0x7ff98c0a2e20).stop 2021-03-09T10:51:11.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800134c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:11.772 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800134c0 unknown :-1 s=CLOSED pgs=9257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:11.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:11.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800139b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:11.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:11.773 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0adc20 2021-03-09T10:51:11.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013ea0 con 0x7ff99c0613a0 2021-03-09T10:51:11.774 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013fe0 con 0x7ff980051540 2021-03-09T10:51:11.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:11.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:11.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:11.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:11.775 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.764+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:11.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.768+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:11.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.768+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:11.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.768+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:11.777 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.768+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:11.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.768+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002fd10 con 0x7ff980051540 2021-03-09T10:51:11.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.768+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800477a0 con 0x7ff980051540 2021-03-09T10:51:11.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.768+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800139b0 secure :-1 s=READY pgs=9269 cs=0 l=1 rev1=1 rx=0x7ff980018360 tx=0x7ff98c0a9250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:11.778 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:11.772+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31467 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:11.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.776+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff980051540 2021-03-09T10:51:11.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff980051540 2021-03-09T10:51:11.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.784+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff980051540 2021-03-09T10:51:11.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.784+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:11.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.784+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:11.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:11.784+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:16.778 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800139b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:16.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800139b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:16.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800139b0 secure :-1 s=READY pgs=9269 cs=0 l=1 rev1=1 rx=0x7ff980018360 tx=0x7ff98c0a9250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:16.779 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800139b0 secure :-1 s=READY pgs=9269 cs=0 l=1 rev1=1 rx=0x7ff980018360 tx=0x7ff98c0a9250).stop 2021-03-09T10:51:16.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800139b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:16.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800139b0 unknown :-1 s=CLOSED pgs=9269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:16.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:16.780 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:16.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:16.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016920 con 0x7ff99c0adc20 2021-03-09T10:51:16.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800147a0 con 0x7ff99c0613a0 2021-03-09T10:51:16.781 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013670 con 0x7ff98000ac10 2021-03-09T10:51:16.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:16.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:16.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:16.782 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:16.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.772+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:16.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.776+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:16.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.776+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:16.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.776+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:16.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.776+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:16.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.776+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800137e0 con 0x7ff98000ac10 2021-03-09T10:51:16.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.776+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002fd10 con 0x7ff98000ac10 2021-03-09T10:51:16.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.776+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013180 secure :-1 s=READY pgs=9283 cs=0 l=1 rev1=1 rx=0x7ff98c052f80 tx=0x7ff98c0a2b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:16.785 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:16.776+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31506 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:16.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c07de90 con 0x7ff98000ac10 2021-03-09T10:51:16.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07dcb0 con 0x7ff98000ac10 2021-03-09T10:51:16.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c07d840 con 0x7ff98000ac10 2021-03-09T10:51:16.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.788+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:16.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.788+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:16.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:16.788+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:21.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:21.783 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:21.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013180 secure :-1 s=READY pgs=9283 cs=0 l=1 rev1=1 rx=0x7ff98c052f80 tx=0x7ff98c0a2b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:21.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013180 secure :-1 s=READY pgs=9283 cs=0 l=1 rev1=1 rx=0x7ff98c052f80 tx=0x7ff98c0a2b10).stop 2021-03-09T10:51:21.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:21.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013180 unknown :-1 s=CLOSED pgs=9283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:21.784 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:21.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:21.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:21.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:21.785 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009b70 con 0x7ff99c0adc20 2021-03-09T10:51:21.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:21.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016920 con 0x7ff99c0613a0 2021-03-09T10:51:21.786 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff980051540 2021-03-09T10:51:21.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:21.787 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:21.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:21.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:21.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:21.788 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:21.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:21.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013e60 con 0x7ff980051540 2021-03-09T10:51:21.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980014140 con 0x7ff980051540 2021-03-09T10:51:21.789 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.780+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013970 secure :-1 s=READY pgs=9289 cs=0 l=1 rev1=1 rx=0x7ff980009800 tx=0x7ff994029c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:21.790 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:21.784+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31542 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:21.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.784+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980051540 2021-03-09T10:51:21.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.784+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081340 con 0x7ff980051540 2021-03-09T10:51:21.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.788+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994075070 con 0x7ff980051540 2021-03-09T10:51:21.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.788+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:21.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.788+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:21.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:21.788+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:26.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:26.790 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:26.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013970 secure :-1 s=READY pgs=9289 cs=0 l=1 rev1=1 rx=0x7ff980009800 tx=0x7ff994029c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:26.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013970 secure :-1 s=READY pgs=9289 cs=0 l=1 rev1=1 rx=0x7ff980009800 tx=0x7ff994029c30).stop 2021-03-09T10:51:26.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:26.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013970 unknown :-1 s=CLOSED pgs=9289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:26.791 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:26.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:26.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:26.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016920 con 0x7ff99c0adc20 2021-03-09T10:51:26.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009b50 con 0x7ff99c0613a0 2021-03-09T10:51:26.792 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002fc20 con 0x7ff98000ac10 2021-03-09T10:51:26.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:26.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:26.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:26.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:26.793 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.783+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:26.794 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.787+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:26.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.787+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:26.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.787+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:26.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.787+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:26.795 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.787+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98002fd60 con 0x7ff98000ac10 2021-03-09T10:51:26.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.787+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800137e0 con 0x7ff98000ac10 2021-03-09T10:51:26.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.787+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9302 cs=0 l=1 rev1=1 rx=0x7ff980007170 tx=0x7ff98c0a2cc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:26.796 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:26.787+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31581 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:26.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.791+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:51:26.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.791+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:51:26.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.795+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:51:26.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.795+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:26.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.795+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:26.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:26.795+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:31.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:31.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:31.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9302 cs=0 l=1 rev1=1 rx=0x7ff980007170 tx=0x7ff98c0a2cc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:31.796 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9302 cs=0 l=1 rev1=1 rx=0x7ff980007170 tx=0x7ff98c0a2cc0).stop 2021-03-09T10:51:31.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:31.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=CLOSED pgs=9302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:31.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:31.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:31.797 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:31.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009b50 con 0x7ff99c0adc20 2021-03-09T10:51:31.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016080 con 0x7ff99c0613a0 2021-03-09T10:51:31.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff980051540 2021-03-09T10:51:31.798 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:31.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:31.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:31.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:31.799 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:31.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:31.800 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:31.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:31.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:31.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013ca0 con 0x7ff980051540 2021-03-09T10:51:31.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98002fd60 con 0x7ff980051540 2021-03-09T10:51:31.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.791+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 secure :-1 s=READY pgs=9311 cs=0 l=1 rev1=1 rx=0x7ff98c018040 tx=0x7ff98c0aa4a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:31.802 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:31.795+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31617 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:31.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.795+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c099120 con 0x7ff980051540 2021-03-09T10:51:31.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.795+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c099ec0 con 0x7ff980051540 2021-03-09T10:51:31.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.799+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00b330 con 0x7ff980051540 2021-03-09T10:51:31.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.799+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:31.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.799+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:31.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:31.799+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:35.689 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:51:35.692 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:51:36.801 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:36.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:36.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 secure :-1 s=READY pgs=9311 cs=0 l=1 rev1=1 rx=0x7ff98c018040 tx=0x7ff98c0aa4a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:36.802 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 secure :-1 s=READY pgs=9311 cs=0 l=1 rev1=1 rx=0x7ff98c018040 tx=0x7ff98c0aa4a0).stop 2021-03-09T10:51:36.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800137b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:36.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 unknown :-1 s=CLOSED pgs=9311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:36.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:36.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:36.803 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:36.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009b50 con 0x7ff99c0adc20 2021-03-09T10:51:36.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800049d0 con 0x7ff99c0613a0 2021-03-09T10:51:36.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017be0 con 0x7ff98000ac10 2021-03-09T10:51:36.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:36.804 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:36.805 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:36.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:36.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.795+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:36.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.799+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:36.806 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.799+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:36.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.799+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:36.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.799+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:36.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.799+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017d20 con 0x7ff98000ac10 2021-03-09T10:51:36.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.799+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013e60 con 0x7ff98000ac10 2021-03-09T10:51:36.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.799+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9321 cs=0 l=1 rev1=1 rx=0x7ff9940780e0 tx=0x7ff994060310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:36.808 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:36.799+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31656 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:36.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.803+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98000ac10 2021-03-09T10:51:36.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.807+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98000ac10 2021-03-09T10:51:36.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.807+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:51:36.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.807+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:36.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.807+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:36.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:36.807+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:41.807 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:41.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:41.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9321 cs=0 l=1 rev1=1 rx=0x7ff9940780e0 tx=0x7ff994060310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:41.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9321 cs=0 l=1 rev1=1 rx=0x7ff9940780e0 tx=0x7ff994060310).stop 2021-03-09T10:51:41.808 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:41.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=CLOSED pgs=9321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:41.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:41.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:41.809 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:41.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009b50 con 0x7ff99c0adc20 2021-03-09T10:51:41.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017be0 con 0x7ff99c0613a0 2021-03-09T10:51:41.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff980051540 2021-03-09T10:51:41.810 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:41.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:41.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:41.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:41.811 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:41.812 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:41.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:41.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:41.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:41.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980056ef0 con 0x7ff980051540 2021-03-09T10:51:41.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017d20 con 0x7ff980051540 2021-03-09T10:51:41.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.803+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 secure :-1 s=READY pgs=9333 cs=0 l=1 rev1=1 rx=0x7ff99400bf80 tx=0x7ff994081670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:41.814 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:41.807+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31692 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:41.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.807+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994094340 con 0x7ff980051540 2021-03-09T10:51:41.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.807+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff980051540 2021-03-09T10:51:41.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.811+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994094130 con 0x7ff980051540 2021-03-09T10:51:41.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.811+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:41.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.811+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:41.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:41.811+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:46.813 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:46.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:46.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 secure :-1 s=READY pgs=9333 cs=0 l=1 rev1=1 rx=0x7ff99400bf80 tx=0x7ff994081670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:46.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 secure :-1 s=READY pgs=9333 cs=0 l=1 rev1=1 rx=0x7ff99400bf80 tx=0x7ff994081670).stop 2021-03-09T10:51:46.814 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800137b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:46.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 unknown :-1 s=CLOSED pgs=9333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:46.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:46.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:46.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:46.815 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009b50 con 0x7ff99c0adc20 2021-03-09T10:51:46.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0613a0 2021-03-09T10:51:46.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017be0 con 0x7ff98000ac10 2021-03-09T10:51:46.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:46.816 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:46.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:46.817 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:46.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.807+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:46.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.811+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:46.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.811+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:46.818 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.811+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:46.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.811+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:46.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.811+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980017d20 con 0x7ff98000ac10 2021-03-09T10:51:46.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.811+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980056ef0 con 0x7ff98000ac10 2021-03-09T10:51:46.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.811+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9346 cs=0 l=1 rev1=1 rx=0x7ff99409bfb0 tx=0x7ff99400c410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:46.820 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:46.811+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31731 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:46.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.815+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff98000ac10 2021-03-09T10:51:46.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.815+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400c790 con 0x7ff98000ac10 2021-03-09T10:51:46.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.819+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994094130 con 0x7ff98000ac10 2021-03-09T10:51:46.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.819+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:46.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.819+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:46.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:46.819+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:51.819 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:51.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:51.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9346 cs=0 l=1 rev1=1 rx=0x7ff99409bfb0 tx=0x7ff99400c410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:51.820 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9346 cs=0 l=1 rev1=1 rx=0x7ff99409bfb0 tx=0x7ff99400c410).stop 2021-03-09T10:51:51.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:51.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=CLOSED pgs=9346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:51.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:51.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:51.821 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:51.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0adc20 2021-03-09T10:51:51.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056f40 con 0x7ff99c0613a0 2021-03-09T10:51:51.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057080 con 0x7ff980051540 2021-03-09T10:51:51.822 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.811+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:51.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.815+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:51.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.815+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:51.823 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.815+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:51.824 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.815+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:51.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.815+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:51.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.815+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:51.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.815+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:51.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.815+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:51.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.815+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015070 con 0x7ff980051540 2021-03-09T10:51:51.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.815+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980017d20 con 0x7ff980051540 2021-03-09T10:51:51.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.819+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 secure :-1 s=READY pgs=9355 cs=0 l=1 rev1=1 rx=0x7ff994011520 tx=0x7ff9940a3bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:51.827 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:51.819+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31767 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:51.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.819+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980051540 2021-03-09T10:51:51.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400ae20 con 0x7ff980051540 2021-03-09T10:51:51.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:51:51.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.827+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:51.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.827+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:51.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:51.827+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:51:56.825 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:56.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:56.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 secure :-1 s=READY pgs=9355 cs=0 l=1 rev1=1 rx=0x7ff994011520 tx=0x7ff9940a3bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:56.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 secure :-1 s=READY pgs=9355 cs=0 l=1 rev1=1 rx=0x7ff994011520 tx=0x7ff9940a3bc0).stop 2021-03-09T10:51:56.826 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800137b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:56.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800137b0 unknown :-1 s=CLOSED pgs=9355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:56.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:56.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:56.827 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:56.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800551b0 con 0x7ff99c0adc20 2021-03-09T10:51:56.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017fd0 con 0x7ff99c0613a0 2021-03-09T10:51:56.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056f40 con 0x7ff98000ac10 2021-03-09T10:51:56.828 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:51:56.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:56.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:51:56.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:56.829 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.819+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:56.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.823+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:56.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.823+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:56.830 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.823+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:56.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.823+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:56.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.823+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047600 con 0x7ff98000ac10 2021-03-09T10:51:56.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.823+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015070 con 0x7ff98000ac10 2021-03-09T10:51:56.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.823+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9366 cs=0 l=1 rev1=1 rx=0x7ff980049d20 tx=0x7ff9940ab4e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:56.832 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:51:56.823+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31806 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:51:56.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940ab260 con 0x7ff98000ac10 2021-03-09T10:51:56.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940ab260 con 0x7ff98000ac10 2021-03-09T10:51:56.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:51:56.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.831+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:56.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.831+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:56.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:51:56.831+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:01.831 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:52:01.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:01.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9366 cs=0 l=1 rev1=1 rx=0x7ff980049d20 tx=0x7ff9940ab4e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:01.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9366 cs=0 l=1 rev1=1 rx=0x7ff980049d20 tx=0x7ff9940ab4e0).stop 2021-03-09T10:52:01.832 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:01.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=CLOSED pgs=9366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:01.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:01.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:01.833 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:01.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015070 con 0x7ff99c0adc20 2021-03-09T10:52:01.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0613a0 2021-03-09T10:52:01.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056f90 con 0x7ff980051540 2021-03-09T10:52:01.834 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:01.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:01.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:01.835 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:01.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:01.836 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:01.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:01.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:01.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:01.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047790 con 0x7ff980051540 2021-03-09T10:52:01.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047600 con 0x7ff980051540 2021-03-09T10:52:01.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.827+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 secure :-1 s=READY pgs=9375 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994081750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:01.838 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:01.831+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31842 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:01.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff980051540 2021-03-09T10:52:01.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098ce0 con 0x7ff980051540 2021-03-09T10:52:01.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.835+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:52:01.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.835+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:01.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.835+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:01.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:01.835+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:05.727 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:52:05.730 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:52:06.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:06.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:06.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 secure :-1 s=READY pgs=9375 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994081750).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:06.837 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 secure :-1 s=READY pgs=9375 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994081750).stop 2021-03-09T10:52:06.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:06.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 unknown :-1 s=CLOSED pgs=9375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:06.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:06.838 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:06.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:06.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047920 con 0x7ff99c0adc20 2021-03-09T10:52:06.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015070 con 0x7ff99c0613a0 2021-03-09T10:52:06.839 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047600 con 0x7ff98000ac10 2021-03-09T10:52:06.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:06.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:06.840 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:06.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:06.841 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.831+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:06.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.835+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:06.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.835+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:06.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.835+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:06.842 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.835+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:06.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.835+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980039270 con 0x7ff98000ac10 2021-03-09T10:52:06.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.835+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013ca0 con 0x7ff98000ac10 2021-03-09T10:52:06.843 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:06.835+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31881 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:06.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.835+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9390 cs=0 l=1 rev1=1 rx=0x7ff98c079860 tx=0x7ff98c00ba80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:06.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.839+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c022ed0 con 0x7ff98000ac10 2021-03-09T10:52:06.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.839+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c022cf0 con 0x7ff98000ac10 2021-03-09T10:52:06.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:52:06.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.843+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:06.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.843+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:06.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:06.843+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:11.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:11.843 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:11.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9390 cs=0 l=1 rev1=1 rx=0x7ff98c079860 tx=0x7ff98c00ba80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:11.844 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9390 cs=0 l=1 rev1=1 rx=0x7ff98c079860 tx=0x7ff98c00ba80).stop 2021-03-09T10:52:11.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:11.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=CLOSED pgs=9390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:11.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:11.845 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:11.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:11.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c640 con 0x7ff99c0adc20 2021-03-09T10:52:11.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047920 con 0x7ff99c0613a0 2021-03-09T10:52:11.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015070 con 0x7ff980051540 2021-03-09T10:52:11.846 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:11.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:11.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:11.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:11.847 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:11.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:11.848 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:11.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:11.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:11.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047600 con 0x7ff980051540 2021-03-09T10:52:11.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.839+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047790 con 0x7ff980051540 2021-03-09T10:52:11.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.843+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 secure :-1 s=READY pgs=9400 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff99400df10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:11.850 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:11.843+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31917 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:11.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b11c0 con 0x7ff980051540 2021-03-09T10:52:11.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99402a180 con 0x7ff980051540 2021-03-09T10:52:11.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:52:11.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.851+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:11.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.851+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:11.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:11.851+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:16.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:16.849 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:16.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 secure :-1 s=READY pgs=9400 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff99400df10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:16.850 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 secure :-1 s=READY pgs=9400 cs=0 l=1 rev1=1 rx=0x7ff994036d20 tx=0x7ff99400df10).stop 2021-03-09T10:52:16.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:16.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 unknown :-1 s=CLOSED pgs=9400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:16.851 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:16.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:16.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:16.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047950 con 0x7ff99c0adc20 2021-03-09T10:52:16.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c640 con 0x7ff99c0613a0 2021-03-09T10:52:16.852 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015070 con 0x7ff98000ac10 2021-03-09T10:52:16.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:16.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:16.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:16.853 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:16.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.843+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:16.854 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.847+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:16.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.847+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:16.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.847+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:16.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.847+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:16.855 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.847+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022ff0 con 0x7ff98000ac10 2021-03-09T10:52:16.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.847+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980039270 con 0x7ff98000ac10 2021-03-09T10:52:16.856 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:16.847+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31956 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:16.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.847+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9410 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c090000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:16.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0aa1d0 con 0x7ff98000ac10 2021-03-09T10:52:16.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00a000 con 0x7ff98000ac10 2021-03-09T10:52:16.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:52:16.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.855+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:16.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.855+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:16.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:16.855+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:21.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:21.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:21.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9410 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c090000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:21.856 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 secure :-1 s=READY pgs=9410 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c090000).stop 2021-03-09T10:52:21.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:21.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013160 unknown :-1 s=CLOSED pgs=9410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:21.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:21.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:21.857 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:21.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c640 con 0x7ff99c0adc20 2021-03-09T10:52:21.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005ec0 con 0x7ff99c0613a0 2021-03-09T10:52:21.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047930 con 0x7ff980051540 2021-03-09T10:52:21.858 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:21.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:21.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:21.859 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:21.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:21.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:21.860 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:21.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:21.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:21.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015070 con 0x7ff980051540 2021-03-09T10:52:21.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.851+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047600 con 0x7ff980051540 2021-03-09T10:52:21.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 secure :-1 s=READY pgs=9419 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff994060230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:21.862 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:21.855+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31992 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:21.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99402a180 con 0x7ff980051540 2021-03-09T10:52:21.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994095300 con 0x7ff980051540 2021-03-09T10:52:21.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940a3cd0 con 0x7ff980051540 2021-03-09T10:52:21.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.859+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:21.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.859+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:21.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:21.859+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:26.861 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:26.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:26.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 secure :-1 s=READY pgs=9419 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff994060230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:26.862 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 secure :-1 s=READY pgs=9419 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff994060230).stop 2021-03-09T10:52:26.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:26.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013790 unknown :-1 s=CLOSED pgs=9419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:26.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:26.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:26.863 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:26.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:26.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:26.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c640 con 0x7ff99c0adc20 2021-03-09T10:52:26.864 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023180 con 0x7ff99c0613a0 2021-03-09T10:52:26.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800232c0 con 0x7ff98000ac10 2021-03-09T10:52:26.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:26.865 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:26.866 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.855+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:26.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:26.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:26.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:26.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:26.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980039410 con 0x7ff98000ac10 2021-03-09T10:52:26.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015070 con 0x7ff98000ac10 2021-03-09T10:52:26.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013130 secure :-1 s=READY pgs=9431 cs=0 l=1 rev1=1 rx=0x7ff994095290 tx=0x7ff994095370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:26.869 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:26.859+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32031 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:26.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095510 con 0x7ff98000ac10 2021-03-09T10:52:26.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.863+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:52:26.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:52:26.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.867+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:26.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.867+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:26.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:26.867+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:31.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:31.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:31.867 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013130 secure :-1 s=READY pgs=9431 cs=0 l=1 rev1=1 rx=0x7ff994095290 tx=0x7ff994095370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:31.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013130 secure :-1 s=READY pgs=9431 cs=0 l=1 rev1=1 rx=0x7ff994095290 tx=0x7ff994095370).stop 2021-03-09T10:52:31.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:31.868 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013130 unknown :-1 s=CLOSED pgs=9431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:31.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:31.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:31.869 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:31.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015070 con 0x7ff99c0adc20 2021-03-09T10:52:31.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0613a0 2021-03-09T10:52:31.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.859+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013c00 con 0x7ff980051540 2021-03-09T10:52:31.870 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.863+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:31.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.863+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:31.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.863+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:31.871 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:31.872 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.863+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:31.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:31.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:31.873 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:31.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:31.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800269c0 con 0x7ff980051540 2021-03-09T10:52:31.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980039410 con 0x7ff980051540 2021-03-09T10:52:31.875 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:31.867+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32067 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:31.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013710 secure :-1 s=READY pgs=9445 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99408b690).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:31.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405fb90 con 0x7ff980051540 2021-03-09T10:52:31.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405f9b0 con 0x7ff980051540 2021-03-09T10:52:31.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:52:31.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.871+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:31.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.875+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:31.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:31.875+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:35.781 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:52:35.784 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:52:36.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:36.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:36.874 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013710 secure :-1 s=READY pgs=9445 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99408b690).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:36.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013710 secure :-1 s=READY pgs=9445 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99408b690).stop 2021-03-09T10:52:36.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:36.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013710 unknown :-1 s=CLOSED pgs=9445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:36.875 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:36.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:36.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:36.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015070 con 0x7ff99c0adc20 2021-03-09T10:52:36.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023180 con 0x7ff99c0613a0 2021-03-09T10:52:36.876 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800232c0 con 0x7ff98000ac10 2021-03-09T10:52:36.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:36.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:36.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:36.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:36.877 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:36.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:36.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:36.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:36.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:36.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047600 con 0x7ff98000ac10 2021-03-09T10:52:36.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.867+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800269c0 con 0x7ff98000ac10 2021-03-09T10:52:36.880 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:36.871+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32105 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:36.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 secure :-1 s=READY pgs=9449 cs=0 l=1 rev1=1 rx=0x7ff9940603a0 tx=0x7ff994001990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:36.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994001e60 con 0x7ff98000ac10 2021-03-09T10:52:36.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.875+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940b11c0 con 0x7ff98000ac10 2021-03-09T10:52:36.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.879+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:52:36.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.879+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:36.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.879+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:36.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:36.879+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:41.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800130f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:41.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800130f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:41.878 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 secure :-1 s=READY pgs=9449 cs=0 l=1 rev1=1 rx=0x7ff9940603a0 tx=0x7ff994001990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:41.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 secure :-1 s=READY pgs=9449 cs=0 l=1 rev1=1 rx=0x7ff9940603a0 tx=0x7ff994001990).stop 2021-03-09T10:52:41.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800130f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:41.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 unknown :-1 s=CLOSED pgs=9449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:41.879 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:41.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:41.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:41.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:41.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:41.880 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049cc0 con 0x7ff99c0adc20 2021-03-09T10:52:41.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015070 con 0x7ff99c0613a0 2021-03-09T10:52:41.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff980051540 2021-03-09T10:52:41.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:41.881 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:41.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.871+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:41.882 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.875+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:41.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:41.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.875+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:41.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.875+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:41.883 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.875+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013bb0 con 0x7ff980051540 2021-03-09T10:52:41.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.875+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980022ff0 con 0x7ff980051540 2021-03-09T10:52:41.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.879+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800136c0 secure :-1 s=READY pgs=9465 cs=0 l=1 rev1=1 rx=0x7ff98c00b7b0 tx=0x7ff98c00bef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:41.885 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:41.875+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32142 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:41.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.879+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c008b00 con 0x7ff980051540 2021-03-09T10:52:41.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c008b00 con 0x7ff980051540 2021-03-09T10:52:41.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.887+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff980051540 2021-03-09T10:52:41.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.887+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:41.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.887+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:41.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:41.887+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:46.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:46.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:46.884 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800136c0 secure :-1 s=READY pgs=9465 cs=0 l=1 rev1=1 rx=0x7ff98c00b7b0 tx=0x7ff98c00bef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:46.885 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800136c0 secure :-1 s=READY pgs=9465 cs=0 l=1 rev1=1 rx=0x7ff98c00b7b0 tx=0x7ff98c00bef0).stop 2021-03-09T10:52:46.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800136c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:46.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800136c0 unknown :-1 s=CLOSED pgs=9465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:46.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:46.886 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:46.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:46.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015070 con 0x7ff99c0adc20 2021-03-09T10:52:46.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800940c0 con 0x7ff99c0613a0 2021-03-09T10:52:46.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094200 con 0x7ff98000ac10 2021-03-09T10:52:46.887 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:46.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:46.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:46.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:46.888 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.878+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:46.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.882+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:46.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.882+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:46.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.882+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:46.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.882+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:46.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.882+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047790 con 0x7ff98000ac10 2021-03-09T10:52:46.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.882+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047600 con 0x7ff98000ac10 2021-03-09T10:52:46.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.882+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 secure :-1 s=READY pgs=9472 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994078070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:46.891 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:46.882+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32178 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:46.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.882+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400ce80 con 0x7ff98000ac10 2021-03-09T10:52:46.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.882+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400c980 con 0x7ff98000ac10 2021-03-09T10:52:46.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.886+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:52:46.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.886+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:46.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.890+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:46.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:46.890+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:51.889 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800130f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:51.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800130f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:51.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 secure :-1 s=READY pgs=9472 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994078070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:51.890 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 secure :-1 s=READY pgs=9472 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994078070).stop 2021-03-09T10:52:51.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800130f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:51.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800130f0 unknown :-1 s=CLOSED pgs=9472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:51.891 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:51.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:51.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:51.892 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026770 con 0x7ff99c0adc20 2021-03-09T10:52:51.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980094240 con 0x7ff99c0613a0 2021-03-09T10:52:51.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047920 con 0x7ff980051540 2021-03-09T10:52:51.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:51.893 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:51.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:51.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:51.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.882+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:51.894 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.886+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:51.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.886+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:51.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.886+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:51.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.886+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:51.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.886+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015070 con 0x7ff980051540 2021-03-09T10:52:51.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.886+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047790 con 0x7ff980051540 2021-03-09T10:52:51.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.886+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013620 secure :-1 s=READY pgs=9481 cs=0 l=1 rev1=1 rx=0x7ff994034a20 tx=0x7ff99400c980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:51.897 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:51.886+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32217 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:51.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.890+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405ee00 con 0x7ff980051540 2021-03-09T10:52:51.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.890+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994034c20 con 0x7ff980051540 2021-03-09T10:52:51.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:52:51.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.894+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:51.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.894+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:51.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:51.894+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:52:56.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.886+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:56.895 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.886+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:56.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.886+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013620 secure :-1 s=READY pgs=9481 cs=0 l=1 rev1=1 rx=0x7ff994034a20 tx=0x7ff99400c980).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:56.896 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.886+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013620 secure :-1 s=READY pgs=9481 cs=0 l=1 rev1=1 rx=0x7ff994034a20 tx=0x7ff99400c980).stop 2021-03-09T10:52:56.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.886+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:56.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013620 unknown :-1 s=CLOSED pgs=9481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:56.897 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:56.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:56.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:56.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980093180 con 0x7ff99c0adc20 2021-03-09T10:52:56.898 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047930 con 0x7ff99c0613a0 2021-03-09T10:52:56.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980046720 con 0x7ff98000ac10 2021-03-09T10:52:56.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:52:56.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:56.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:52:56.899 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:56.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.890+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:56.900 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:56.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:56.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:56.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:56.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980023190 con 0x7ff98000ac10 2021-03-09T10:52:56.901 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015070 con 0x7ff98000ac10 2021-03-09T10:52:56.902 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:52:56.894+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32253 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:52:56.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013090 secure :-1 s=READY pgs=9497 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99408c410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:56.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408c200 con 0x7ff98000ac10 2021-03-09T10:52:56.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408c200 con 0x7ff98000ac10 2021-03-09T10:52:56.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:52:56.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.902+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:56.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.902+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:56.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:52:56.902+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:01.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:01.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:01.902 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013090 secure :-1 s=READY pgs=9497 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99408c410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:01.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013090 secure :-1 s=READY pgs=9497 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff99408c410).stop 2021-03-09T10:53:01.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:01.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013090 unknown :-1 s=CLOSED pgs=9497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:01.903 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:01.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:01.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:01.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0adc20 2021-03-09T10:53:01.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015070 con 0x7ff99c0613a0 2021-03-09T10:53:01.904 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013580 con 0x7ff980051540 2021-03-09T10:53:01.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:01.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:01.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:01.905 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:01.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.894+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:01.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.898+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:01.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.898+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:01.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.898+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:01.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.898+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:01.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.898+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047600 con 0x7ff980051540 2021-03-09T10:53:01.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.898+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980023190 con 0x7ff980051540 2021-03-09T10:53:01.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.898+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046700 secure :-1 s=READY pgs=9505 cs=0 l=1 rev1=1 rx=0x7ff994095d40 tx=0x7ff99408c200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:01.909 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:01.902+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32292 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:01.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009a20 con 0x7ff980051540 2021-03-09T10:53:01.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009a20 con 0x7ff980051540 2021-03-09T10:53:01.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994095d70 con 0x7ff980051540 2021-03-09T10:53:01.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.906+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:01.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.906+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:01.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:01.906+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:05.854 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:53:05.857 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:53:06.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980046700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:06.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980046700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:06.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046700 secure :-1 s=READY pgs=9505 cs=0 l=1 rev1=1 rx=0x7ff994095d40 tx=0x7ff99408c200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:06.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046700 secure :-1 s=READY pgs=9505 cs=0 l=1 rev1=1 rx=0x7ff994095d40 tx=0x7ff99408c200).stop 2021-03-09T10:53:06.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980046700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:06.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980046700 unknown :-1 s=CLOSED pgs=9505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:06.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:06.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:06.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:06.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000cd30 con 0x7ff99c0adc20 2021-03-09T10:53:06.911 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000ce70 con 0x7ff99c0613a0 2021-03-09T10:53:06.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000cfb0 con 0x7ff98000ac10 2021-03-09T10:53:06.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:06.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:06.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:06.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:06.912 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:06.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:06.913 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:06.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:06.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:06.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013540 con 0x7ff98000ac10 2021-03-09T10:53:06.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.902+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013bb0 con 0x7ff98000ac10 2021-03-09T10:53:06.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.906+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9514 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00baf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:06.915 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:06.906+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32328 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:06.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c07d2e0 con 0x7ff98000ac10 2021-03-09T10:53:06.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07d2e0 con 0x7ff98000ac10 2021-03-09T10:53:06.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:53:06.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.914+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:06.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.914+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:06.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:06.914+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:11.914 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:11.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:11.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9514 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00baf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:11.915 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9514 cs=0 l=1 rev1=1 rx=0x7ff98c03e040 tx=0x7ff98c00baf0).stop 2021-03-09T10:53:11.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:11.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=CLOSED pgs=9514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:11.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:11.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:11.916 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:11.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092430 con 0x7ff99c0adc20 2021-03-09T10:53:11.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0613a0 2021-03-09T10:53:11.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000cce0 con 0x7ff980051540 2021-03-09T10:53:11.917 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:11.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:11.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:11.918 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.906+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:11.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:11.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:11.919 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:11.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:11.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.910+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:11.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000ce20 con 0x7ff980051540 2021-03-09T10:53:11.920 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.910+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047600 con 0x7ff980051540 2021-03-09T10:53:11.921 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:11.914+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32367 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:11.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 secure :-1 s=READY pgs=9528 cs=0 l=1 rev1=1 rx=0x7ff99400e040 tx=0x7ff994078070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:11.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400ae10 con 0x7ff980051540 2021-03-09T10:53:11.925 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400ac30 con 0x7ff980051540 2021-03-09T10:53:11.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:53:11.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.922+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:11.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.926+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:11.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:11.926+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:16.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:16.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:16.921 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 secure :-1 s=READY pgs=9528 cs=0 l=1 rev1=1 rx=0x7ff99400e040 tx=0x7ff994078070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:16.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 secure :-1 s=READY pgs=9528 cs=0 l=1 rev1=1 rx=0x7ff99400e040 tx=0x7ff994078070).stop 2021-03-09T10:53:16.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:16.922 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 unknown :-1 s=CLOSED pgs=9528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:16.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:16.923 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:16.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:16.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:16.924 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:16.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022930 con 0x7ff99c0adc20 2021-03-09T10:53:16.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013c60 con 0x7ff99c0613a0 2021-03-09T10:53:16.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800923e0 con 0x7ff98000ac10 2021-03-09T10:53:16.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:16.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:16.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:16.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:16.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:16.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:16.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:16.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000cc90 con 0x7ff98000ac10 2021-03-09T10:53:16.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.914+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013540 con 0x7ff98000ac10 2021-03-09T10:53:16.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.918+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9536 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c0066c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:16.932 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:16.918+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32403 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:16.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.918+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08dc30 con 0x7ff98000ac10 2021-03-09T10:53:16.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08dc30 con 0x7ff98000ac10 2021-03-09T10:53:16.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c04bb60 con 0x7ff98000ac10 2021-03-09T10:53:16.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.926+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:16.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.926+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:16.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:16.926+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:21.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:21.926 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:21.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9536 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c0066c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:21.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9536 cs=0 l=1 rev1=1 rx=0x7ff98c08dfd0 tx=0x7ff98c0066c0).stop 2021-03-09T10:53:21.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:21.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=CLOSED pgs=9536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:21.927 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:21.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:21.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:21.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0adc20 2021-03-09T10:53:21.928 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013c60 con 0x7ff99c0613a0 2021-03-09T10:53:21.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000cfb0 con 0x7ff980051540 2021-03-09T10:53:21.929 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:21.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:21.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:21.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:21.930 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:21.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:21.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:21.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:21.931 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:21.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800477a0 con 0x7ff980051540 2021-03-09T10:53:21.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000cc90 con 0x7ff980051540 2021-03-09T10:53:21.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.922+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 secure :-1 s=READY pgs=9547 cs=0 l=1 rev1=1 rx=0x7ff98c084450 tx=0x7ff98c08dc30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:21.933 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:21.926+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32442 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:21.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:53:21.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980051540 2021-03-09T10:53:21.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c072b80 con 0x7ff980051540 2021-03-09T10:53:21.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.930+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:21.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.930+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:21.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:21.930+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:26.932 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:26.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:26.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 secure :-1 s=READY pgs=9547 cs=0 l=1 rev1=1 rx=0x7ff98c084450 tx=0x7ff98c08dc30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:26.933 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 secure :-1 s=READY pgs=9547 cs=0 l=1 rev1=1 rx=0x7ff98c084450 tx=0x7ff98c08dc30).stop 2021-03-09T10:53:26.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff980013770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:26.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff980013770 unknown :-1 s=CLOSED pgs=9547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:26.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:26.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:26.934 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:26.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000cb10 con 0x7ff99c0adc20 2021-03-09T10:53:26.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000cc50 con 0x7ff99c0613a0 2021-03-09T10:53:26.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000cfb0 con 0x7ff98000ac10 2021-03-09T10:53:26.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:26.935 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:26.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:26.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:26.936 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:26.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:26.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:26.937 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:26.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:26.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d120 con 0x7ff98000ac10 2021-03-09T10:53:26.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.926+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000ce20 con 0x7ff98000ac10 2021-03-09T10:53:26.938 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.930+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9560 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99400bf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:26.939 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:26.930+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32478 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:26.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.930+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400df10 con 0x7ff98000ac10 2021-03-09T10:53:26.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99402a180 con 0x7ff98000ac10 2021-03-09T10:53:26.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:53:26.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.938+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:26.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.942+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:26.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:26.942+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:31.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:53:31.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:31.939 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9560 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99400bf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:31.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9560 cs=0 l=1 rev1=1 rx=0x7ff99407e190 tx=0x7ff99400bf40).stop 2021-03-09T10:53:31.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:31.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=CLOSED pgs=9560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:31.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:31.940 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:31.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:31.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092db0 con 0x7ff99c0adc20 2021-03-09T10:53:31.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0613a0 2021-03-09T10:53:31.941 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980013770 con 0x7ff980051540 2021-03-09T10:53:31.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:31.942 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:31.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:31.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:31.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:31.943 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:31.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:31.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:31.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:31.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800138b0 con 0x7ff980051540 2021-03-09T10:53:31.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.934+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800477a0 con 0x7ff980051540 2021-03-09T10:53:31.945 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:31.938+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32517 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:31.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000d2b0 secure :-1 s=READY pgs=9569 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c006650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:31.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c09fa80 con 0x7ff980051540 2021-03-09T10:53:31.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c09fa80 con 0x7ff980051540 2021-03-09T10:53:31.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980051540 2021-03-09T10:53:31.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.942+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:31.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.942+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:31.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:31.942+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:36.075 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:53:36.078 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:53:36.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:36.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:36.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000d2b0 secure :-1 s=READY pgs=9569 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c006650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:36.944 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000d2b0 secure :-1 s=READY pgs=9569 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c006650).stop 2021-03-09T10:53:36.945 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff98000d2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:36.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff98000d2b0 unknown :-1 s=CLOSED pgs=9569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:36.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:36.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:36.946 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:36.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d7a0 con 0x7ff99c0adc20 2021-03-09T10:53:36.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800153a0 con 0x7ff99c0613a0 2021-03-09T10:53:36.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800473f0 con 0x7ff98000ac10 2021-03-09T10:53:36.947 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:36.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:36.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:36.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:36.948 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:36.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:36.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:36.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:36.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:36.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047530 con 0x7ff98000ac10 2021-03-09T10:53:36.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.938+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800138b0 con 0x7ff98000ac10 2021-03-09T10:53:36.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.942+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9578 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c08ded0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:36.951 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:36.942+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32553 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:36.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c03c020 con 0x7ff98000ac10 2021-03-09T10:53:36.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.942+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07cb80 con 0x7ff98000ac10 2021-03-09T10:53:36.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c085510 con 0x7ff98000ac10 2021-03-09T10:53:36.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.950+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:36.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.950+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:36.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:36.950+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:41.949 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:53:41.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:41.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9578 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c08ded0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:41.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9578 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c08ded0).stop 2021-03-09T10:53:41.950 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:41.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=CLOSED pgs=9578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:41.951 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:41.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:41.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:41.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800473a0 con 0x7ff99c0adc20 2021-03-09T10:53:41.952 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020820 con 0x7ff99c0613a0 2021-03-09T10:53:41.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d2b0 con 0x7ff980051540 2021-03-09T10:53:41.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:41.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:41.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:41.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:41.953 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:41.954 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:41.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:41.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:41.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:41.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d3f0 con 0x7ff980051540 2021-03-09T10:53:41.955 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.946+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d120 con 0x7ff980051540 2021-03-09T10:53:41.956 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:41.950+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32592 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:41.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800138b0 secure :-1 s=READY pgs=9595 cs=0 l=1 rev1=1 rx=0x7ff99408c4c0 tx=0x7ff99400df80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:41.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99405dc50 con 0x7ff980051540 2021-03-09T10:53:41.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99405dc50 con 0x7ff980051540 2021-03-09T10:53:41.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980051540 2021-03-09T10:53:41.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.958+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:41.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.958+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:41.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:41.958+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:46.956 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:46.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:46.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800138b0 secure :-1 s=READY pgs=9595 cs=0 l=1 rev1=1 rx=0x7ff99408c4c0 tx=0x7ff99400df80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:46.957 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800138b0 secure :-1 s=READY pgs=9595 cs=0 l=1 rev1=1 rx=0x7ff99408c4c0 tx=0x7ff99400df80).stop 2021-03-09T10:53:46.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 msgr2=0x7ff9800138b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:46.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980051540 0x7ff9800138b0 unknown :-1 s=CLOSED pgs=9595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:46.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:46.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:46.958 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:46.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d290 con 0x7ff99c0adc20 2021-03-09T10:53:46.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff99c0613a0 2021-03-09T10:53:46.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800260d0 con 0x7ff98000ac10 2021-03-09T10:53:46.959 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:46.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:46.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:46.960 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:46.961 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.950+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:46.962 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:46.954+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32628 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:46.962 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:46.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:46.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:46.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:46.963 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d580 con 0x7ff98000ac10 2021-03-09T10:53:46.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d3f0 con 0x7ff98000ac10 2021-03-09T10:53:46.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9606 cs=0 l=1 rev1=1 rx=0x7ff9940780e0 tx=0x7ff9940816e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:46.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940833b0 con 0x7ff98000ac10 2021-03-09T10:53:46.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994094130 con 0x7ff98000ac10 2021-03-09T10:53:46.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940831a0 con 0x7ff98000ac10 2021-03-09T10:53:46.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.962+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:46.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.962+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:46.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:46.962+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:49.906 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:53:49.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:49.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7290 cs=0 l=1 rev1=1 rx=0x7ff98407b4a0 tx=0x7ff98407d070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:49.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=7290 cs=0 l=1 rev1=1 rx=0x7ff98407b4a0 tx=0x7ff98407d070).stop 2021-03-09T10:53:49.907 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff99affd700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:53:49.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff9937fe700 4 mgrc ms_handle_reset ms_handle_reset con 0x7ff980083100 2021-03-09T10:53:49.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff9937fe700 4 mgrc reconnect Terminating session with v2:172.21.15.52:6808/14433 2021-03-09T10:53:49.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:49.908 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=7290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:49.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff9937fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:49.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 0x7ff9800138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:49.909 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 0x7ff9800138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:49.910 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:49.898+0000 7ff99affd700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] conn(0x7ff980051540 0x7ff9800138b0 secure :-1 s=READY pgs=9099 cs=0 l=1 rev1=1 rx=0x7ff984049060 tx=0x7ff98404f000).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:51.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:51.964 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:51.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9606 cs=0 l=1 rev1=1 rx=0x7ff9940780e0 tx=0x7ff9940816e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:51.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 secure :-1 s=READY pgs=9606 cs=0 l=1 rev1=1 rx=0x7ff9940780e0 tx=0x7ff9940816e0).stop 2021-03-09T10:53:51.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980013050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:51.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980013050 unknown :-1 s=CLOSED pgs=9606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:51.965 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:51.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:51.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:51.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d430 con 0x7ff99c0adc20 2021-03-09T10:53:51.966 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.954+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800279a0 con 0x7ff99c0613a0 2021-03-09T10:53:51.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980018110 con 0x7ff980083100 2021-03-09T10:53:51.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:51.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:51.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:51.967 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:51.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:51.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:51.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:51.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:51.968 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:51.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980013da0 con 0x7ff980083100 2021-03-09T10:53:51.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047530 con 0x7ff980083100 2021-03-09T10:53:51.969 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.958+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9609 cs=0 l=1 rev1=1 rx=0x7ff98c03c290 tx=0x7ff98c00ba80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:51.970 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:51.962+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32667 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:51.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04c8d0 con 0x7ff980083100 2021-03-09T10:53:51.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c04c8d0 con 0x7ff980083100 2021-03-09T10:53:51.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.966+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980083100 2021-03-09T10:53:51.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.966+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:51.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.966+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:51.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:51.966+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:53:56.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:56.970 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:56.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9609 cs=0 l=1 rev1=1 rx=0x7ff98c03c290 tx=0x7ff98c00ba80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:56.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9609 cs=0 l=1 rev1=1 rx=0x7ff98c03c290 tx=0x7ff98c00ba80).stop 2021-03-09T10:53:56.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:56.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:56.971 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980012e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:56.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:56.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:56.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003690 con 0x7ff99c0adc20 2021-03-09T10:53:56.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d430 con 0x7ff99c0613a0 2021-03-09T10:53:56.972 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800209c0 con 0x7ff98000ac10 2021-03-09T10:53:56.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:53:56.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:56.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:53:56.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:56.973 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.962+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980012e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:56.974 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.966+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:56.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.966+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:56.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.966+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:56.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.966+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:56.975 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.966+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800476d0 con 0x7ff98000ac10 2021-03-09T10:53:56.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.966+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d580 con 0x7ff98000ac10 2021-03-09T10:53:56.976 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.966+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980012e90 secure :-1 s=READY pgs=9622 cs=0 l=1 rev1=1 rx=0x7ff98008ece0 tx=0x7ff9940b1fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:56.976 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:53:56.970+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32703 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:53:56.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.970+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b3c0 con 0x7ff98000ac10 2021-03-09T10:53:56.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.978+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:53:56.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.978+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff9940604b0 con 0x7ff98000ac10 2021-03-09T10:53:56.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.978+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:56.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.978+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:56.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:53:56.978+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:01.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980012e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:01.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980012e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:01.977 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980012e90 secure :-1 s=READY pgs=9622 cs=0 l=1 rev1=1 rx=0x7ff98008ece0 tx=0x7ff9940b1fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:01.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980012e90 secure :-1 s=READY pgs=9622 cs=0 l=1 rev1=1 rx=0x7ff98008ece0 tx=0x7ff9940b1fd0).stop 2021-03-09T10:54:01.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980012e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:01.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980012e90 unknown :-1 s=CLOSED pgs=9622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:01.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:01.978 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:01.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:01.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:01.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:01.979 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980003830 con 0x7ff99c0adc20 2021-03-09T10:54:01.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800290c0 con 0x7ff99c0613a0 2021-03-09T10:54:01.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800182b0 con 0x7ff980083100 2021-03-09T10:54:01.980 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:01.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:01.981 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:01.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:01.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:01.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:01.982 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:01.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980020b60 con 0x7ff980083100 2021-03-09T10:54:01.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800476d0 con 0x7ff980083100 2021-03-09T10:54:01.983 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:01.974+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32742 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:01.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.974+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 secure :-1 s=READY pgs=9633 cs=0 l=1 rev1=1 rx=0x7ff994068470 tx=0x7ff994031cc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:01.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.982+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:54:01.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.982+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940b1d80 con 0x7ff980083100 2021-03-09T10:54:01.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.982+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994094130 con 0x7ff980083100 2021-03-09T10:54:01.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.982+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:01.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.982+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:01.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:01.986+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:06.115 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:54:06.118 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:54:06.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:06.983 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.973+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:06.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 secure :-1 s=READY pgs=9633 cs=0 l=1 rev1=1 rx=0x7ff994068470 tx=0x7ff994031cc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:06.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.973+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 secure :-1 s=READY pgs=9633 cs=0 l=1 rev1=1 rx=0x7ff994068470 tx=0x7ff994031cc0).stop 2021-03-09T10:54:06.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.973+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:06.984 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.973+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 unknown :-1 s=CLOSED pgs=9633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:06.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.973+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:06.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:06.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:06.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016200 con 0x7ff99c0adc20 2021-03-09T10:54:06.985 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029280 con 0x7ff99c0613a0 2021-03-09T10:54:06.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:06.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023570 con 0x7ff98000ac10 2021-03-09T10:54:06.986 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:06.987 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:06.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:06.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:06.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:06.988 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:06.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:06.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:06.989 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:54:06.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013da0 con 0x7ff98000ac10 2021-03-09T10:54:06.990 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.977+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9641 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c0797e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:06.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c045d50 con 0x7ff98000ac10 2021-03-09T10:54:06.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c045eb0 con 0x7ff98000ac10 2021-03-09T10:54:06.991 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:06.981+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32778 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:06.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.985+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:54:06.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.985+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:06.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.989+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:06.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:06.989+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:11.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:11.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:11.991 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9641 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c0797e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:11.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9641 cs=0 l=1 rev1=1 rx=0x7ff98c0a3110 tx=0x7ff98c0797e0).stop 2021-03-09T10:54:11.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:11.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:11.992 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:11.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:11.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:11.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023570 con 0x7ff99c0adc20 2021-03-09T10:54:11.993 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026300 con 0x7ff99c0613a0 2021-03-09T10:54:11.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800182b0 con 0x7ff980083100 2021-03-09T10:54:11.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:11.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:11.994 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:11.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:11.995 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.981+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:11.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:11.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:11.996 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:11.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:11.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019c40 con 0x7ff980083100 2021-03-09T10:54:11.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.985+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980020b60 con 0x7ff980083100 2021-03-09T10:54:11.997 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.985+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012b00 secure :-1 s=READY pgs=9653 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99402aea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:11.998 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:11.989+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32817 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:11.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020ea0 con 0x7ff980083100 2021-03-09T10:54:12.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:54:12.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:54:12.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.997+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:12.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.997+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:12.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:11.997+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:16.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:16.998 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:16.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012b00 secure :-1 s=READY pgs=9653 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99402aea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:16.999 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012b00 secure :-1 s=READY pgs=9653 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99402aea0).stop 2021-03-09T10:54:17.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980012b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:17.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980012b00 unknown :-1 s=CLOSED pgs=9653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:17.000 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:17.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:17.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:17.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980023570 con 0x7ff99c0adc20 2021-03-09T10:54:17.001 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026300 con 0x7ff99c0613a0 2021-03-09T10:54:17.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800279c0 con 0x7ff98000ac10 2021-03-09T10:54:17.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:17.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:17.002 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:17.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:17.003 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:17.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:17.004 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:17.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:17.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:17.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800476d0 con 0x7ff98000ac10 2021-03-09T10:54:17.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980013da0 con 0x7ff98000ac10 2021-03-09T10:54:17.005 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.993+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9663 cs=0 l=1 rev1=1 rx=0x7ff98c00b7b0 tx=0x7ff98c035180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:17.006 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:16.997+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32853 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:17.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff98000ac10 2021-03-09T10:54:17.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:16.997+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff98000ac10 2021-03-09T10:54:17.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:17.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:54:17.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:17.001+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:17.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:17.001+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:17.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:17.001+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:22.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:22.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:22.006 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9663 cs=0 l=1 rev1=1 rx=0x7ff98c00b7b0 tx=0x7ff98c035180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:22.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9663 cs=0 l=1 rev1=1 rx=0x7ff98c00b7b0 tx=0x7ff98c035180).stop 2021-03-09T10:54:22.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:22.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:22.007 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:22.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:22.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:22.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800182b0 con 0x7ff99c0adc20 2021-03-09T10:54:22.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026300 con 0x7ff99c0613a0 2021-03-09T10:54:22.008 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092ac0 con 0x7ff980083100 2021-03-09T10:54:22.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:22.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:22.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:22.009 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:22.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:22.010 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:22.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:22.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:22.011 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:22.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980012a60 con 0x7ff980083100 2021-03-09T10:54:22.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800476d0 con 0x7ff980083100 2021-03-09T10:54:22.012 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.001+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d2b0 secure :-1 s=READY pgs=9674 cs=0 l=1 rev1=1 rx=0x7ff98c006aa0 tx=0x7ff98c020c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:22.013 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:22.005+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32892 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:22.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.005+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c045e40 con 0x7ff980083100 2021-03-09T10:54:22.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c045e40 con 0x7ff980083100 2021-03-09T10:54:22.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0840e0 con 0x7ff980083100 2021-03-09T10:54:22.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.009+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:22.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.013+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:22.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:22.013+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:27.013 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:54:27.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:27.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d2b0 secure :-1 s=READY pgs=9674 cs=0 l=1 rev1=1 rx=0x7ff98c006aa0 tx=0x7ff98c020c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:27.014 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d2b0 secure :-1 s=READY pgs=9674 cs=0 l=1 rev1=1 rx=0x7ff98c006aa0 tx=0x7ff98c020c60).stop 2021-03-09T10:54:27.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:27.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d2b0 unknown :-1 s=CLOSED pgs=9674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:27.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:27.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:27.015 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:27.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800182b0 con 0x7ff99c0adc20 2021-03-09T10:54:27.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026300 con 0x7ff99c0613a0 2021-03-09T10:54:27.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800279c0 con 0x7ff98000ac10 2021-03-09T10:54:27.016 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:27.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:27.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:27.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:27.017 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:27.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:27.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:27.018 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:27.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:27.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800476d0 con 0x7ff98000ac10 2021-03-09T10:54:27.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980012a60 con 0x7ff98000ac10 2021-03-09T10:54:27.019 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.009+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9685 cs=0 l=1 rev1=1 rx=0x7ff9800912d0 tx=0x7ff98c017410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:27.020 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:27.013+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32928 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:27.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.013+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c045cf0 con 0x7ff98000ac10 2021-03-09T10:54:27.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.013+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c045e50 con 0x7ff98000ac10 2021-03-09T10:54:27.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c045cf0 con 0x7ff98000ac10 2021-03-09T10:54:27.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.017+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:27.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.017+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:27.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:27.017+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:32.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:32.020 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:32.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9685 cs=0 l=1 rev1=1 rx=0x7ff9800912d0 tx=0x7ff98c017410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:32.021 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9685 cs=0 l=1 rev1=1 rx=0x7ff9800912d0 tx=0x7ff98c017410).stop 2021-03-09T10:54:32.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:32.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:32.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:32.022 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:32.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:32.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800182b0 con 0x7ff99c0adc20 2021-03-09T10:54:32.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026300 con 0x7ff99c0613a0 2021-03-09T10:54:32.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d5e0 con 0x7ff980083100 2021-03-09T10:54:32.023 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:32.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:32.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:32.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:32.024 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:32.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:32.025 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:32.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:32.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:32.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980012a20 con 0x7ff980083100 2021-03-09T10:54:32.026 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019c40 con 0x7ff980083100 2021-03-09T10:54:32.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.017+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 secure :-1 s=READY pgs=9695 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994031da0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:32.027 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:32.017+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32966 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:32.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.021+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020ea0 con 0x7ff980083100 2021-03-09T10:54:32.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.025+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994020220 con 0x7ff980083100 2021-03-09T10:54:32.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:54:32.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.029+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:32.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.029+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:32.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:32.029+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:36.161 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:54:36.164 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:54:37.027 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:54:37.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:37.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 secure :-1 s=READY pgs=9695 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994031da0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:37.028 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 secure :-1 s=READY pgs=9695 cs=0 l=1 rev1=1 rx=0x7ff99407e000 tx=0x7ff994031da0).stop 2021-03-09T10:54:37.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:37.029 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0f0 unknown :-1 s=CLOSED pgs=9695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:37.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:37.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:37.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:37.030 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800182b0 con 0x7ff99c0adc20 2021-03-09T10:54:37.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026300 con 0x7ff99c0613a0 2021-03-09T10:54:37.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:54:37.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:37.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:37.031 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:37.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:37.032 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:37.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:37.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:37.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:37.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:37.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800968b0 con 0x7ff98000ac10 2021-03-09T10:54:37.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.025+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980012a20 con 0x7ff98000ac10 2021-03-09T10:54:37.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9707 cs=0 l=1 rev1=1 rx=0x7ff9800908c0 tx=0x7ff994032010).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:37.035 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:37.029+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33003 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:37.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99406d400 con 0x7ff98000ac10 2021-03-09T10:54:37.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff98000ac10 2021-03-09T10:54:37.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99406d1f0 con 0x7ff98000ac10 2021-03-09T10:54:37.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.037+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:37.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.037+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:37.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:37.037+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:42.033 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:42.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:42.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9707 cs=0 l=1 rev1=1 rx=0x7ff9800908c0 tx=0x7ff994032010).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:42.034 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9707 cs=0 l=1 rev1=1 rx=0x7ff9800908c0 tx=0x7ff994032010).stop 2021-03-09T10:54:42.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:42.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:42.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:42.035 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:42.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:42.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800182b0 con 0x7ff99c0adc20 2021-03-09T10:54:42.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026300 con 0x7ff99c0613a0 2021-03-09T10:54:42.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980048310 con 0x7ff980083100 2021-03-09T10:54:42.036 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:42.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:42.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:42.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:42.037 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.029+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:42.038 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:42.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:42.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:42.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:42.039 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d5c0 con 0x7ff980083100 2021-03-09T10:54:42.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.033+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800476d0 con 0x7ff980083100 2021-03-09T10:54:42.040 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.033+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0d0 secure :-1 s=READY pgs=9717 cs=0 l=1 rev1=1 rx=0x7ff98c017d30 tx=0x7ff98c085f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:42.040 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:42.037+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33041 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:42.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c099120 con 0x7ff980083100 2021-03-09T10:54:42.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0528c0 con 0x7ff980083100 2021-03-09T10:54:42.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.041+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:54:42.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.041+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:42.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.041+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:42.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:42.041+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:47.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:47.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:47.041 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0d0 secure :-1 s=READY pgs=9717 cs=0 l=1 rev1=1 rx=0x7ff98c017d30 tx=0x7ff98c085f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:47.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0d0 secure :-1 s=READY pgs=9717 cs=0 l=1 rev1=1 rx=0x7ff98c017d30 tx=0x7ff98c085f90).stop 2021-03-09T10:54:47.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98000d0d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:47.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98000d0d0 unknown :-1 s=CLOSED pgs=9717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:47.042 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:47.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:47.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:47.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:47.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800182b0 con 0x7ff99c0adc20 2021-03-09T10:54:47.043 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026300 con 0x7ff99c0613a0 2021-03-09T10:54:47.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800476f0 con 0x7ff98000ac10 2021-03-09T10:54:47.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:47.044 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:47.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:47.045 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:47.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:47.046 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:47.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:47.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:47.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:54:47.047 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.037+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d5c0 con 0x7ff98000ac10 2021-03-09T10:54:47.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.041+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9727 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c03d220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:47.048 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:47.041+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33078 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:47.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.041+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c099120 con 0x7ff98000ac10 2021-03-09T10:54:47.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.041+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff98000ac10 2021-03-09T10:54:47.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.049+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c020cd0 con 0x7ff98000ac10 2021-03-09T10:54:47.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.049+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:47.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.049+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:47.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:47.049+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:52.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.041+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:52.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.041+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:52.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.041+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9727 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c03d220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:52.048 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.041+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9727 cs=0 l=1 rev1=1 rx=0x7ff98c0a2d90 tx=0x7ff98c03d220).stop 2021-03-09T10:54:52.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.041+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:52.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.041+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:52.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.041+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:52.049 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.041+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:52.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.041+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:52.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.041+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800182b0 con 0x7ff99c0adc20 2021-03-09T10:54:52.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980029c30 con 0x7ff99c0613a0 2021-03-09T10:54:52.050 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800926d0 con 0x7ff980083100 2021-03-09T10:54:52.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:52.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:52.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:52.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:52.051 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:52.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:52.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:52.052 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:52.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:52.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047930 con 0x7ff980083100 2021-03-09T10:54:52.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800968b0 con 0x7ff980083100 2021-03-09T10:54:52.053 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.045+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096a40 secure :-1 s=READY pgs=9737 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99409b240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:52.054 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:52.049+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33116 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:52.054 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.049+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020220 con 0x7ff980083100 2021-03-09T10:54:52.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.049+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994020ea0 con 0x7ff980083100 2021-03-09T10:54:52.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.053+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980083100 2021-03-09T10:54:52.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.053+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:52.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.053+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:52.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:52.053+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:54:57.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:54:57.055 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:57.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096a40 secure :-1 s=READY pgs=9737 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99409b240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:57.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096a40 secure :-1 s=READY pgs=9737 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff99409b240).stop 2021-03-09T10:54:57.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:57.056 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096a40 unknown :-1 s=CLOSED pgs=9737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:57.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:57.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:57.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:57.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005930 con 0x7ff99c0adc20 2021-03-09T10:54:57.057 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:54:57.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800182b0 con 0x7ff99c0613a0 2021-03-09T10:54:57.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026300 con 0x7ff98000ac10 2021-03-09T10:54:57.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:57.058 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:54:57.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.049+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:57.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.053+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:57.059 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.053+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:57.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.053+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:57.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.053+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:57.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.053+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:57.060 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.053+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800476d0 con 0x7ff98000ac10 2021-03-09T10:54:57.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.053+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047930 con 0x7ff98000ac10 2021-03-09T10:54:57.061 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.053+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9750 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99408d260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:57.061 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:54:57.057+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33153 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:54:57.062 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff98000ac10 2021-03-09T10:54:57.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.061+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:54:57.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff98000ac10 2021-03-09T10:54:57.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.065+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:57.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.069+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:57.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:54:57.069+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:02.064 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:02.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:02.065 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9750 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99408d260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:02.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9750 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff99408d260).stop 2021-03-09T10:55:02.066 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:02.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:02.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:02.067 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:02.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:02.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026300 con 0x7ff99c0adc20 2021-03-09T10:55:02.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057150 con 0x7ff99c0613a0 2021-03-09T10:55:02.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d750 con 0x7ff980083100 2021-03-09T10:55:02.068 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:02.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:02.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:02.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:02.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.057+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:02.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.061+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:02.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.061+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:02.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.061+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:02.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.061+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:02.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.061+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047930 con 0x7ff980083100 2021-03-09T10:55:02.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.061+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d5c0 con 0x7ff980083100 2021-03-09T10:55:02.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.061+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800968b0 secure :-1 s=READY pgs=9759 cs=0 l=1 rev1=1 rx=0x7ff98c04c000 tx=0x7ff98c0797f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:02.072 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:02.065+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33191 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:02.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:55:02.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2e20 con 0x7ff980083100 2021-03-09T10:55:02.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.069+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c017120 con 0x7ff980083100 2021-03-09T10:55:02.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.069+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:02.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.069+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:02.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:02.069+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:06.319 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:55:06.329 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:55:07.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:07.069 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:07.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800968b0 secure :-1 s=READY pgs=9759 cs=0 l=1 rev1=1 rx=0x7ff98c04c000 tx=0x7ff98c0797f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:07.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800968b0 secure :-1 s=READY pgs=9759 cs=0 l=1 rev1=1 rx=0x7ff98c04c000 tx=0x7ff98c0797f0).stop 2021-03-09T10:55:07.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800968b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:07.070 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800968b0 unknown :-1 s=CLOSED pgs=9759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:07.071 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:07.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:07.072 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:07.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d760 con 0x7ff99c0adc20 2021-03-09T10:55:07.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800572f0 con 0x7ff99c0613a0 2021-03-09T10:55:07.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049300 con 0x7ff98000ac10 2021-03-09T10:55:07.073 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:07.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:07.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:07.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:07.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.065+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:07.074 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.069+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:07.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.069+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:07.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.069+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:07.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.069+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:07.075 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.069+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980018260 con 0x7ff98000ac10 2021-03-09T10:55:07.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.069+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800476d0 con 0x7ff98000ac10 2021-03-09T10:55:07.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.069+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9769 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff994031ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:07.076 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:07.073+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33228 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:07.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400bed0 con 0x7ff98000ac10 2021-03-09T10:55:07.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400bed0 con 0x7ff98000ac10 2021-03-09T10:55:07.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.077+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:55:07.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.077+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:07.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.077+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:07.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:07.077+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:12.076 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:12.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:12.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9769 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff994031ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:12.077 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 secure :-1 s=READY pgs=9769 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff994031ed0).stop 2021-03-09T10:55:12.078 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:12.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:12.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:12.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:12.079 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:12.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980011ea0 con 0x7ff99c0adc20 2021-03-09T10:55:12.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800476d0 con 0x7ff99c0613a0 2021-03-09T10:55:12.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800055a0 con 0x7ff980083100 2021-03-09T10:55:12.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:12.080 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:12.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:12.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:12.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:12.081 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:12.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:12.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:12.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:12.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005710 con 0x7ff980083100 2021-03-09T10:55:12.082 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.073+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047930 con 0x7ff980083100 2021-03-09T10:55:12.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.077+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800498e0 secure :-1 s=READY pgs=9781 cs=0 l=1 rev1=1 rx=0x7ff9800136f0 tx=0x7ff98c0a2e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:12.084 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:12.077+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33266 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:12.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.081+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980083100 2021-03-09T10:55:12.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.081+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a2c00 con 0x7ff980083100 2021-03-09T10:55:12.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.085+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980083100 2021-03-09T10:55:12.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.085+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:12.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.085+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:12.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:12.085+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:17.083 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.077+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800498e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:55:17.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.077+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800498e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:17.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800498e0 secure :-1 s=READY pgs=9781 cs=0 l=1 rev1=1 rx=0x7ff9800136f0 tx=0x7ff98c0a2e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:17.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800498e0 secure :-1 s=READY pgs=9781 cs=0 l=1 rev1=1 rx=0x7ff9800136f0 tx=0x7ff98c0a2e20).stop 2021-03-09T10:55:17.084 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800498e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:17.085 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800498e0 unknown :-1 s=CLOSED pgs=9781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:17.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:17.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:17.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:17.086 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056f90 con 0x7ff99c0adc20 2021-03-09T10:55:17.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:17.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800055a0 con 0x7ff99c0613a0 2021-03-09T10:55:17.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980047930 con 0x7ff98000ac10 2021-03-09T10:55:17.087 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:17.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:17.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:17.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:17.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:17.088 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:17.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:17.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:17.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:55:17.089 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005710 con 0x7ff98000ac10 2021-03-09T10:55:17.090 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.081+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 secure :-1 s=READY pgs=9791 cs=0 l=1 rev1=1 rx=0x7ff98c092060 tx=0x7ff98c0a6070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:17.090 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:17.085+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33303 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:17.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.085+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c072c50 con 0x7ff98000ac10 2021-03-09T10:55:17.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.085+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c072c50 con 0x7ff98000ac10 2021-03-09T10:55:17.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.089+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff98000ac10 2021-03-09T10:55:17.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.089+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:17.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.093+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:17.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:17.093+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:22.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:22.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:22.091 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 secure :-1 s=READY pgs=9791 cs=0 l=1 rev1=1 rx=0x7ff98c092060 tx=0x7ff98c0a6070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:22.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 secure :-1 s=READY pgs=9791 cs=0 l=1 rev1=1 rx=0x7ff98c092060 tx=0x7ff98c0a6070).stop 2021-03-09T10:55:22.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800058a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:22.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 unknown :-1 s=CLOSED pgs=9791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:22.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:22.092 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:22.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:22.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800199c0 con 0x7ff99c0adc20 2021-03-09T10:55:22.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049cc0 con 0x7ff99c0613a0 2021-03-09T10:55:22.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800055a0 con 0x7ff980083100 2021-03-09T10:55:22.093 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.085+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:22.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.089+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:22.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.089+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:22.094 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.089+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:22.095 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.089+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:22.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.089+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:22.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.089+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:22.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.089+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:22.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.089+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:22.096 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.089+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047930 con 0x7ff980083100 2021-03-09T10:55:22.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.089+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980018260 con 0x7ff980083100 2021-03-09T10:55:22.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.093+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9803 cs=0 l=1 rev1=1 rx=0x7ff9940296a0 tx=0x7ff994031f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:22.098 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:22.093+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33341 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:22.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.093+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:55:22.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.097+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:55:22.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.097+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980083100 2021-03-09T10:55:22.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.097+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:22.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.101+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:22.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:22.101+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:27.097 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:27.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:27.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9803 cs=0 l=1 rev1=1 rx=0x7ff9940296a0 tx=0x7ff994031f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:27.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9803 cs=0 l=1 rev1=1 rx=0x7ff9940296a0 tx=0x7ff994031f40).stop 2021-03-09T10:55:27.098 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:27.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:27.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:27.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:27.099 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:27.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980049e60 con 0x7ff99c0adc20 2021-03-09T10:55:27.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800055a0 con 0x7ff99c0613a0 2021-03-09T10:55:27.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:55:27.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:27.100 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:27.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:27.101 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:27.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.093+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:27.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.097+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:27.102 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.097+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:27.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.097+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:27.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.097+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:27.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.097+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980018060 con 0x7ff98000ac10 2021-03-09T10:55:27.103 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.097+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047930 con 0x7ff98000ac10 2021-03-09T10:55:27.104 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.097+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 secure :-1 s=READY pgs=9814 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940342b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:27.104 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:27.101+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33378 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:27.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.101+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994031000 con 0x7ff98000ac10 2021-03-09T10:55:27.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.105+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:55:27.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.109+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:55:27.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.109+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:27.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.109+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:27.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:27.109+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:32.105 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:32.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:32.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 secure :-1 s=READY pgs=9814 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940342b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:32.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 secure :-1 s=READY pgs=9814 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff9940342b0).stop 2021-03-09T10:55:32.106 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800058a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:32.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 unknown :-1 s=CLOSED pgs=9814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:32.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:32.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:32.107 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:32.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:32.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980017020 con 0x7ff99c0adc20 2021-03-09T10:55:32.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:32.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000db10 con 0x7ff99c0613a0 2021-03-09T10:55:32.108 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005580 con 0x7ff980083100 2021-03-09T10:55:32.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:32.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:32.109 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:32.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:32.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:32.110 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:32.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:32.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.100+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98004b670 con 0x7ff980083100 2021-03-09T10:55:32.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.104+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980018060 con 0x7ff980083100 2021-03-09T10:55:32.111 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.104+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9823 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994098770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:32.112 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:32.108+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33416 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:32.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:55:32.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994020220 con 0x7ff980083100 2021-03-09T10:55:32.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:55:32.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.112+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:32.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.112+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:32.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:32.112+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:36.368 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:55:36.371 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:55:37.112 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:37.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:37.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9823 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994098770).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:37.113 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9823 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994098770).stop 2021-03-09T10:55:37.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:37.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:37.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:37.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:37.114 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:37.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004d180 con 0x7ff99c0adc20 2021-03-09T10:55:37.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b9a0 con 0x7ff99c0613a0 2021-03-09T10:55:37.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005580 con 0x7ff98000ac10 2021-03-09T10:55:37.115 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:37.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:37.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:37.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:37.116 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:37.117 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:37.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:37.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:37.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:37.118 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:55:37.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.108+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004b670 con 0x7ff98000ac10 2021-03-09T10:55:37.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 secure :-1 s=READY pgs=9834 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99408c100).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:37.120 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:37.112+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33453 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:37.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994098590 con 0x7ff98000ac10 2021-03-09T10:55:37.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.116+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098590 con 0x7ff98000ac10 2021-03-09T10:55:37.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.120+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:55:37.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.120+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:37.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.120+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:37.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:37.120+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:42.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:42.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:42.119 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 secure :-1 s=READY pgs=9834 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99408c100).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:42.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 secure :-1 s=READY pgs=9834 cs=0 l=1 rev1=1 rx=0x7ff994006d80 tx=0x7ff99408c100).stop 2021-03-09T10:55:42.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800058a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:42.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800058a0 unknown :-1 s=CLOSED pgs=9834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:42.120 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:42.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:42.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980046e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:42.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b9a0 con 0x7ff99c0adc20 2021-03-09T10:55:42.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005580 con 0x7ff99c0613a0 2021-03-09T10:55:42.121 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff980083100 2021-03-09T10:55:42.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:42.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:42.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:42.122 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:42.123 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.112+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980046e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:42.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.116+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:42.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.116+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:42.124 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.116+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:42.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.116+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:42.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.116+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980047380 con 0x7ff980083100 2021-03-09T10:55:42.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.116+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98004b670 con 0x7ff980083100 2021-03-09T10:55:42.125 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.116+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980046e90 secure :-1 s=READY pgs=9846 cs=0 l=1 rev1=1 rx=0x7ff99405ec00 tx=0x7ff9940602a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:42.126 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:42.120+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33491 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:42.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.124+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:55:42.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.128+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994034000 con 0x7ff980083100 2021-03-09T10:55:42.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.128+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:55:42.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.128+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:42.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.128+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:42.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:42.128+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:47.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980046e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:47.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980046e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:47.126 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980046e90 secure :-1 s=READY pgs=9846 cs=0 l=1 rev1=1 rx=0x7ff99405ec00 tx=0x7ff9940602a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:47.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980046e90 secure :-1 s=READY pgs=9846 cs=0 l=1 rev1=1 rx=0x7ff99405ec00 tx=0x7ff9940602a0).stop 2021-03-09T10:55:47.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980046e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:47.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980046e90 unknown :-1 s=CLOSED pgs=9846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:47.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:47.127 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:47.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:47.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:55:47.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b9c0 con 0x7ff99c0613a0 2021-03-09T10:55:47.128 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005580 con 0x7ff98000ac10 2021-03-09T10:55:47.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:47.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:47.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:47.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:47.129 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.120+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:47.130 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.124+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:47.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.124+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:47.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.124+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:47.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.124+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:47.131 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.124+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:55:47.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.124+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980047380 con 0x7ff98000ac10 2021-03-09T10:55:47.132 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.124+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b4d0 secure :-1 s=READY pgs=9855 cs=0 l=1 rev1=1 rx=0x7ff980089b40 tx=0x7ff9940a3f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:47.132 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:47.128+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33528 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:47.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.128+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400c150 con 0x7ff98000ac10 2021-03-09T10:55:47.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.136+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994034000 con 0x7ff98000ac10 2021-03-09T10:55:47.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.136+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:55:47.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.136+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:47.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.140+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:47.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:47.140+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:52.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98004b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:52.133 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98004b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:52.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b4d0 secure :-1 s=READY pgs=9855 cs=0 l=1 rev1=1 rx=0x7ff980089b40 tx=0x7ff9940a3f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:52.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b4d0 secure :-1 s=READY pgs=9855 cs=0 l=1 rev1=1 rx=0x7ff980089b40 tx=0x7ff9940a3f00).stop 2021-03-09T10:55:52.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98004b4d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:52.134 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b4d0 unknown :-1 s=CLOSED pgs=9855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:52.135 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:52.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:52.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:52.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:55:52.136 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b9c0 con 0x7ff99c0613a0 2021-03-09T10:55:52.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005a60 con 0x7ff980083100 2021-03-09T10:55:52.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:52.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:52.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:52.137 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.128+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:52.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.132+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:52.138 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.132+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:52.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.132+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:52.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.132+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:52.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.132+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:52.139 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.132+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005580 con 0x7ff980083100 2021-03-09T10:55:52.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.132+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980025f90 con 0x7ff980083100 2021-03-09T10:55:52.140 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.136+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9870 cs=0 l=1 rev1=1 rx=0x7ff994016580 tx=0x7ff9940aae20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:52.141 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:52.136+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33566 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:52.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.136+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:55:52.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.140+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff980083100 2021-03-09T10:55:52.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.140+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:55:52.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.140+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:52.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.144+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:52.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:52.144+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:55:57.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:57.141 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:57.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9870 cs=0 l=1 rev1=1 rx=0x7ff994016580 tx=0x7ff9940aae20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:57.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 secure :-1 s=READY pgs=9870 cs=0 l=1 rev1=1 rx=0x7ff994016580 tx=0x7ff9940aae20).stop 2021-03-09T10:55:57.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800963c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:57.142 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800963c0 unknown :-1 s=CLOSED pgs=9870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:57.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:57.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:57.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:57.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:55:57.143 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e540 con 0x7ff99c0613a0 2021-03-09T10:55:57.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b950 con 0x7ff98000ac10 2021-03-09T10:55:57.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:55:57.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:57.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:55:57.144 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:57.145 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.136+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:57.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.140+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:57.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.140+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:57.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.140+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:57.146 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.140+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:57.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.140+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:55:57.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.140+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005710 con 0x7ff98000ac10 2021-03-09T10:55:57.147 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.140+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b460 secure :-1 s=READY pgs=9880 cs=0 l=1 rev1=1 rx=0x7ff98c02d050 tx=0x7ff98c0a60e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:57.148 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:55:57.140+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33603 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:55:57.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.144+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:55:57.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.148+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:55:57.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:55:57.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.152+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:57.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.152+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:57.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:55:57.152+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:02.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98004b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:02.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98004b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:02.148 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b460 secure :-1 s=READY pgs=9880 cs=0 l=1 rev1=1 rx=0x7ff98c02d050 tx=0x7ff98c0a60e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:02.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b460 secure :-1 s=READY pgs=9880 cs=0 l=1 rev1=1 rx=0x7ff98c02d050 tx=0x7ff98c0a60e0).stop 2021-03-09T10:56:02.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98004b460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:02.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98004b460 unknown :-1 s=CLOSED pgs=9880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:02.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:02.149 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980054e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:02.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:02.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:56:02.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e540 con 0x7ff99c0613a0 2021-03-09T10:56:02.150 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800963c0 con 0x7ff980083100 2021-03-09T10:56:02.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:02.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:02.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:02.151 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:02.152 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980054e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:02.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:02.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:02.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:02.153 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:02.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096500 con 0x7ff980083100 2021-03-09T10:56:02.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.144+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980025f90 con 0x7ff980083100 2021-03-09T10:56:02.154 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.148+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980054e40 secure :-1 s=READY pgs=9887 cs=0 l=1 rev1=1 rx=0x7ff98c0a91c0 tx=0x7ff98c006650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:02.155 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:02.148+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33641 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:02.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.148+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00b330 con 0x7ff980083100 2021-03-09T10:56:02.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.148+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00b490 con 0x7ff980083100 2021-03-09T10:56:02.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a72f0 con 0x7ff980083100 2021-03-09T10:56:02.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.152+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:02.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.156+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:02.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:02.156+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:06.453 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:56:06.456 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:56:07.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:07.155 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980054e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:07.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980054e40 secure :-1 s=READY pgs=9887 cs=0 l=1 rev1=1 rx=0x7ff98c0a91c0 tx=0x7ff98c006650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:07.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980054e40 secure :-1 s=READY pgs=9887 cs=0 l=1 rev1=1 rx=0x7ff98c0a91c0 tx=0x7ff98c006650).stop 2021-03-09T10:56:07.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980054e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:07.156 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980054e40 unknown :-1 s=CLOSED pgs=9887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:07.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:07.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:07.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:07.157 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:56:07.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800963c0 con 0x7ff99c0613a0 2021-03-09T10:56:07.158 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e540 con 0x7ff98000ac10 2021-03-09T10:56:07.159 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:07.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:07.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:07.160 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:07.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:07.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:07.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:07.161 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:07.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:07.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:56:07.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005580 con 0x7ff98000ac10 2021-03-09T10:56:07.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.152+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 secure :-1 s=READY pgs=9897 cs=0 l=1 rev1=1 rx=0x7ff99400e600 tx=0x7ff994020220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:07.163 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:07.156+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33678 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:07.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400e260 con 0x7ff98000ac10 2021-03-09T10:56:07.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:56:07.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.160+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:56:07.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.160+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:07.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.160+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:07.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:07.160+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:12.162 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:12.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:12.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 secure :-1 s=READY pgs=9897 cs=0 l=1 rev1=1 rx=0x7ff99400e600 tx=0x7ff994020220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:12.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 secure :-1 s=READY pgs=9897 cs=0 l=1 rev1=1 rx=0x7ff99400e600 tx=0x7ff994020220).stop 2021-03-09T10:56:12.163 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:12.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 unknown :-1 s=CLOSED pgs=9897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:12.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:12.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:12.164 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:12.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:56:12.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980092680 con 0x7ff99c0613a0 2021-03-09T10:56:12.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800159d0 con 0x7ff980083100 2021-03-09T10:56:12.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:12.165 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:12.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:12.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:12.166 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.156+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:12.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.160+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:12.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.160+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:12.167 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.160+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:12.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.160+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:12.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.160+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800239b0 con 0x7ff980083100 2021-03-09T10:56:12.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.160+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980025f90 con 0x7ff980083100 2021-03-09T10:56:12.168 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.160+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096b80 secure :-1 s=READY pgs=9912 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940689d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:12.169 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:12.164+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33716 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:12.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994062e40 con 0x7ff980083100 2021-03-09T10:56:12.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.168+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980083100 2021-03-09T10:56:12.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:56:12.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.172+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:12.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.172+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:12.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:12.172+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:17.170 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:17.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:17.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096b80 secure :-1 s=READY pgs=9912 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940689d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:17.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096b80 secure :-1 s=READY pgs=9912 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff9940689d0).stop 2021-03-09T10:56:17.171 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:17.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096b80 unknown :-1 s=CLOSED pgs=9912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:17.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:17.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:17.172 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:17.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:56:17.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800963a0 con 0x7ff99c0613a0 2021-03-09T10:56:17.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800034f0 con 0x7ff98000ac10 2021-03-09T10:56:17.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:17.173 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:17.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:17.174 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:17.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.164+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:17.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.168+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:17.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.168+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:17.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.168+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:17.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.168+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:17.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.168+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005720 con 0x7ff98000ac10 2021-03-09T10:56:17.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.168+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800239b0 con 0x7ff98000ac10 2021-03-09T10:56:17.177 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:17.168+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33753 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:17.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 secure :-1 s=READY pgs=9923 cs=0 l=1 rev1=1 rx=0x7ff99409e060 tx=0x7ff994095200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:17.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994014ab0 con 0x7ff98000ac10 2021-03-09T10:56:17.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994014ab0 con 0x7ff98000ac10 2021-03-09T10:56:17.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.180+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994016210 con 0x7ff98000ac10 2021-03-09T10:56:17.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.180+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:17.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.180+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:17.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:17.180+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:22.175 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:22.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:22.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 secure :-1 s=READY pgs=9923 cs=0 l=1 rev1=1 rx=0x7ff99409e060 tx=0x7ff994095200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:22.176 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 secure :-1 s=READY pgs=9923 cs=0 l=1 rev1=1 rx=0x7ff99409e060 tx=0x7ff994095200).stop 2021-03-09T10:56:22.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:22.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 unknown :-1 s=CLOSED pgs=9923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:22.177 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:22.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:22.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:22.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:56:22.178 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008e750 con 0x7ff99c0613a0 2021-03-09T10:56:22.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015130 con 0x7ff980083100 2021-03-09T10:56:22.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:22.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:22.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:22.179 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:22.180 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:22.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:22.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:22.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:22.181 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:22.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980006b00 con 0x7ff980083100 2021-03-09T10:56:22.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005720 con 0x7ff980083100 2021-03-09T10:56:22.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.172+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096010 secure :-1 s=READY pgs=9929 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff99409b240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:22.183 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:22.176+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33791 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:22.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994060230 con 0x7ff980083100 2021-03-09T10:56:22.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994082280 con 0x7ff980083100 2021-03-09T10:56:22.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.180+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994068d90 con 0x7ff980083100 2021-03-09T10:56:22.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.180+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:22.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.184+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:22.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:22.184+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:27.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:27.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:27.182 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096010 secure :-1 s=READY pgs=9929 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff99409b240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:27.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096010 secure :-1 s=READY pgs=9929 cs=0 l=1 rev1=1 rx=0x7ff994001030 tx=0x7ff99409b240).stop 2021-03-09T10:56:27.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:27.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096010 unknown :-1 s=CLOSED pgs=9929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:27.183 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:27.184 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:27.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:27.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:56:27.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ebd0 con 0x7ff99c0613a0 2021-03-09T10:56:27.185 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002de90 con 0x7ff98000ac10 2021-03-09T10:56:27.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:27.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:27.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:27.186 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:27.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.176+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:27.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.180+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:27.187 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.180+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:27.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.180+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:27.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.180+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:27.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.180+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008d2f0 con 0x7ff98000ac10 2021-03-09T10:56:27.188 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.180+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980006b00 con 0x7ff98000ac10 2021-03-09T10:56:27.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.180+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 secure :-1 s=READY pgs=9941 cs=0 l=1 rev1=1 rx=0x7ff99409e060 tx=0x7ff994085260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:27.189 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:27.184+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33827 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:27.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.184+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020ea0 con 0x7ff98000ac10 2021-03-09T10:56:27.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.188+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400cd10 con 0x7ff98000ac10 2021-03-09T10:56:27.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.192+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:56:27.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.192+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:27.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.192+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:27.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:27.192+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:32.189 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:32.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:32.190 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 secure :-1 s=READY pgs=9941 cs=0 l=1 rev1=1 rx=0x7ff99409e060 tx=0x7ff994085260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:32.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 secure :-1 s=READY pgs=9941 cs=0 l=1 rev1=1 rx=0x7ff99409e060 tx=0x7ff994085260).stop 2021-03-09T10:56:32.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:32.191 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096690 unknown :-1 s=CLOSED pgs=9941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:32.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:32.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:32.192 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980095f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:32.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:56:32.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800278c0 con 0x7ff99c0613a0 2021-03-09T10:56:32.193 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ebd0 con 0x7ff980083100 2021-03-09T10:56:32.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:32.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:32.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:32.194 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:32.195 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.184+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980095f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:32.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.188+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:32.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.188+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:32.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.188+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:32.196 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.188+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:32.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.188+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980019d70 con 0x7ff980083100 2021-03-09T10:56:32.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.188+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008d2f0 con 0x7ff980083100 2021-03-09T10:56:32.197 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.188+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980095f70 secure :-1 s=READY pgs=9954 cs=0 l=1 rev1=1 rx=0x7ff98004a540 tx=0x7ff994020220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:32.197 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:32.192+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33866 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:32.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.192+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff980083100 2021-03-09T10:56:32.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.200+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098120 con 0x7ff980083100 2021-03-09T10:56:32.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.200+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:56:32.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.200+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:32.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.200+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:32.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:32.200+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:36.491 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:56:36.505 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:56:37.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980095f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:37.198 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980095f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:37.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980095f70 secure :-1 s=READY pgs=9954 cs=0 l=1 rev1=1 rx=0x7ff98004a540 tx=0x7ff994020220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:37.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980095f70 secure :-1 s=READY pgs=9954 cs=0 l=1 rev1=1 rx=0x7ff98004a540 tx=0x7ff994020220).stop 2021-03-09T10:56:37.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980095f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:37.199 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980095f70 unknown :-1 s=CLOSED pgs=9954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:37.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:37.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:37.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:37.200 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:56:37.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800278c0 con 0x7ff99c0613a0 2021-03-09T10:56:37.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006430 con 0x7ff98000ac10 2021-03-09T10:56:37.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:37.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:37.201 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:37.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:37.202 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.192+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:37.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.196+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:37.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.196+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:37.203 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.196+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:37.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.196+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:37.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.196+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:56:37.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.196+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019d70 con 0x7ff98000ac10 2021-03-09T10:56:37.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.196+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 secure :-1 s=READY pgs=9963 cs=0 l=1 rev1=1 rx=0x7ff9940844f0 tx=0x7ff99407efe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:37.205 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:37.200+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33902 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:37.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.200+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400a250 con 0x7ff98000ac10 2021-03-09T10:56:37.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.200+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400a070 con 0x7ff98000ac10 2021-03-09T10:56:37.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.208+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:56:37.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.208+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:37.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.208+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:37.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:37.208+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:42.204 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800159d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:42.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800159d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:42.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 secure :-1 s=READY pgs=9963 cs=0 l=1 rev1=1 rx=0x7ff9940844f0 tx=0x7ff99407efe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:42.205 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 secure :-1 s=READY pgs=9963 cs=0 l=1 rev1=1 rx=0x7ff9940844f0 tx=0x7ff99407efe0).stop 2021-03-09T10:56:42.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800159d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:42.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 unknown :-1 s=CLOSED pgs=9963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:42.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:42.206 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:42.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:42.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004a020 con 0x7ff99c0adc20 2021-03-09T10:56:42.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d660 con 0x7ff99c0613a0 2021-03-09T10:56:42.207 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800278c0 con 0x7ff980083100 2021-03-09T10:56:42.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:42.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:42.209 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:42.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:42.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:42.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:42.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:42.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:42.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.200+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:42.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.204+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000d7e0 con 0x7ff980083100 2021-03-09T10:56:42.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.204+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096500 con 0x7ff980083100 2021-03-09T10:56:42.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.204+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096690 secure :-1 s=READY pgs=9974 cs=0 l=1 rev1=1 rx=0x7ff98c0052d0 tx=0x7ff98c0a30d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:42.213 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:42.204+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33941 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:42.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.208+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980083100 2021-03-09T10:56:42.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.208+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff980083100 2021-03-09T10:56:42.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.216+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff980083100 2021-03-09T10:56:42.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.216+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:42.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.216+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:42.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:42.216+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:47.210 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:47.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:47.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096690 secure :-1 s=READY pgs=9974 cs=0 l=1 rev1=1 rx=0x7ff98c0052d0 tx=0x7ff98c0a30d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:47.211 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096690 secure :-1 s=READY pgs=9974 cs=0 l=1 rev1=1 rx=0x7ff98c0052d0 tx=0x7ff98c0a30d0).stop 2021-03-09T10:56:47.212 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:47.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096690 unknown :-1 s=CLOSED pgs=9974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:47.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:47.213 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:47.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:47.214 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d660 con 0x7ff99c0adc20 2021-03-09T10:56:47.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dcd0 con 0x7ff99c0613a0 2021-03-09T10:56:47.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:47.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006430 con 0x7ff98000ac10 2021-03-09T10:56:47.215 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.204+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:47.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.208+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:47.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.208+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:47.216 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.208+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:47.217 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.208+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:47.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.208+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:47.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.208+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:47.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.208+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:47.218 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.208+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800267a0 con 0x7ff98000ac10 2021-03-09T10:56:47.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.208+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000d7e0 con 0x7ff98000ac10 2021-03-09T10:56:47.219 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:47.212+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33977 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:47.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.212+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 secure :-1 s=READY pgs=9987 cs=0 l=1 rev1=1 rx=0x7ff98c00c2f0 tx=0x7ff98c00b800).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:47.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.216+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08fcf0 con 0x7ff98000ac10 2021-03-09T10:56:47.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.220+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c09f170 con 0x7ff98000ac10 2021-03-09T10:56:47.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.220+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:56:47.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.220+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:47.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.224+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:47.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:47.224+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:52.219 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800159d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:52.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800159d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:52.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 secure :-1 s=READY pgs=9987 cs=0 l=1 rev1=1 rx=0x7ff98c00c2f0 tx=0x7ff98c00b800).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:52.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 secure :-1 s=READY pgs=9987 cs=0 l=1 rev1=1 rx=0x7ff98c00c2f0 tx=0x7ff98c00b800).stop 2021-03-09T10:56:52.220 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800159d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:52.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 unknown :-1 s=CLOSED pgs=9987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:52.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:52.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:52.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:52.221 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dcd0 con 0x7ff99c0adc20 2021-03-09T10:56:52.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006430 con 0x7ff99c0613a0 2021-03-09T10:56:52.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800278c0 con 0x7ff980083100 2021-03-09T10:56:52.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:52.222 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:52.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:52.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:52.223 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.212+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:52.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.216+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:52.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.216+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:52.224 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.216+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:52.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.216+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:52.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.216+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096980 con 0x7ff980083100 2021-03-09T10:56:52.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.216+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980019d70 con 0x7ff980083100 2021-03-09T10:56:52.225 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.216+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 secure :-1 s=READY pgs=9993 cs=0 l=1 rev1=1 rx=0x7ff99408e5b0 tx=0x7ff994038f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:52.226 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:52.220+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34016 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:52.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.220+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020ea0 con 0x7ff980083100 2021-03-09T10:56:52.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.220+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff980083100 2021-03-09T10:56:52.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.224+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:56:52.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.224+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:52.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.224+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:52.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:52.224+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:56:57.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:56:57.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:57.226 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 secure :-1 s=READY pgs=9993 cs=0 l=1 rev1=1 rx=0x7ff99408e5b0 tx=0x7ff994038f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:57.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 secure :-1 s=READY pgs=9993 cs=0 l=1 rev1=1 rx=0x7ff99408e5b0 tx=0x7ff994038f00).stop 2021-03-09T10:56:57.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:57.227 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 unknown :-1 s=CLOSED pgs=9993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:57.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:57.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:57.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:57.228 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002dcd0 con 0x7ff99c0adc20 2021-03-09T10:56:57.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006430 con 0x7ff99c0613a0 2021-03-09T10:56:57.229 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800278c0 con 0x7ff98000ac10 2021-03-09T10:56:57.230 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:57.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:56:57.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:57.231 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:56:57.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.219+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:57.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.223+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:57.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.223+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:57.232 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.223+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:57.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.223+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:57.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.223+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:56:57.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.223+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800267a0 con 0x7ff98000ac10 2021-03-09T10:56:57.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.223+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 secure :-1 s=READY pgs=10005 cs=0 l=1 rev1=1 rx=0x7ff98000dc00 tx=0x7ff98c00bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:57.234 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:56:57.227+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34052 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:56:57.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.227+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c079700 con 0x7ff98000ac10 2021-03-09T10:56:57.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.227+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c009540 con 0x7ff98000ac10 2021-03-09T10:56:57.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.231+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:56:57.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.231+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:57.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.235+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:57.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:56:57.235+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:02.233 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800159d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:02.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800159d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:02.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 secure :-1 s=READY pgs=10005 cs=0 l=1 rev1=1 rx=0x7ff98000dc00 tx=0x7ff98c00bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:02.234 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 secure :-1 s=READY pgs=10005 cs=0 l=1 rev1=1 rx=0x7ff98000dc00 tx=0x7ff98c00bfd0).stop 2021-03-09T10:57:02.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800159d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:02.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800159d0 unknown :-1 s=CLOSED pgs=10005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:02.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:02.235 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:02.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:02.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006430 con 0x7ff99c0adc20 2021-03-09T10:57:02.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015ec0 con 0x7ff99c0613a0 2021-03-09T10:57:02.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ebd0 con 0x7ff980083100 2021-03-09T10:57:02.236 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:02.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:02.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:02.237 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:02.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.227+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:02.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.231+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:02.238 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.231+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:02.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.231+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:02.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.231+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:02.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.231+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980025f90 con 0x7ff980083100 2021-03-09T10:57:02.239 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.231+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096980 con 0x7ff980083100 2021-03-09T10:57:02.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.231+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 secure :-1 s=READY pgs=10019 cs=0 l=1 rev1=1 rx=0x7ff99408c530 tx=0x7ff994002240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:02.240 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:02.235+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34091 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:02.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.235+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994095ce0 con 0x7ff980083100 2021-03-09T10:57:02.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.239+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994020ea0 con 0x7ff980083100 2021-03-09T10:57:02.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.243+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:57:02.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.243+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:02.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.243+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:02.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:02.243+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:06.638 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:57:06.641 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:57:07.240 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:57:07.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:07.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 secure :-1 s=READY pgs=10019 cs=0 l=1 rev1=1 rx=0x7ff99408c530 tx=0x7ff994002240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:07.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 secure :-1 s=READY pgs=10019 cs=0 l=1 rev1=1 rx=0x7ff99408c530 tx=0x7ff994002240).stop 2021-03-09T10:57:07.241 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980096490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:07.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980096490 unknown :-1 s=CLOSED pgs=10019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:07.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:07.242 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:07.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:07.243 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006430 con 0x7ff99c0adc20 2021-03-09T10:57:07.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ebd0 con 0x7ff99c0613a0 2021-03-09T10:57:07.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:57:07.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:07.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:07.244 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:07.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:07.245 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.235+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:07.246 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:07.239+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34127 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:07.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.239+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:07.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.239+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:07.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.239+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:07.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.239+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:07.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.239+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800158f0 con 0x7ff98000ac10 2021-03-09T10:57:07.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.239+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096980 con 0x7ff98000ac10 2021-03-09T10:57:07.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.239+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199c0 secure :-1 s=READY pgs=10032 cs=0 l=1 rev1=1 rx=0x7ff9800152d0 tx=0x7ff99400c230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:07.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.243+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:57:07.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.247+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994068130 con 0x7ff98000ac10 2021-03-09T10:57:07.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.247+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:57:07.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.247+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:07.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.247+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:07.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:07.247+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:12.246 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800199c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:57:12.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800199c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:12.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199c0 secure :-1 s=READY pgs=10032 cs=0 l=1 rev1=1 rx=0x7ff9800152d0 tx=0x7ff99400c230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:12.247 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199c0 secure :-1 s=READY pgs=10032 cs=0 l=1 rev1=1 rx=0x7ff9800152d0 tx=0x7ff99400c230).stop 2021-03-09T10:57:12.248 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff9800199c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:12.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff9800199c0 unknown :-1 s=CLOSED pgs=10032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:12.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:12.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:12.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:12.249 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056da0 con 0x7ff99c0adc20 2021-03-09T10:57:12.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006430 con 0x7ff99c0613a0 2021-03-09T10:57:12.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ea30 con 0x7ff980083100 2021-03-09T10:57:12.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:12.250 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:12.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:12.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:12.251 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:12.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:12.252 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:12.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:12.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:12.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008eba0 con 0x7ff980083100 2021-03-09T10:57:12.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800158f0 con 0x7ff980083100 2021-03-09T10:57:12.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.243+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 secure :-1 s=READY pgs=10038 cs=0 l=1 rev1=1 rx=0x7ff994034340 tx=0x7ff9940aaf00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:12.254 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:12.247+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34166 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:12.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.251+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994029120 con 0x7ff980083100 2021-03-09T10:57:12.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.251+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098ce0 con 0x7ff980083100 2021-03-09T10:57:12.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.251+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:57:12.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.251+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:12.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.251+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:12.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:12.251+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:17.253 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:57:17.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:17.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 secure :-1 s=READY pgs=10038 cs=0 l=1 rev1=1 rx=0x7ff994034340 tx=0x7ff9940aaf00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:17.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 secure :-1 s=READY pgs=10038 cs=0 l=1 rev1=1 rx=0x7ff994034340 tx=0x7ff9940aaf00).stop 2021-03-09T10:57:17.254 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:17.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 unknown :-1 s=CLOSED pgs=10038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:17.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:17.255 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:17.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:17.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980004a80 con 0x7ff99c0adc20 2021-03-09T10:57:17.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006430 con 0x7ff99c0613a0 2021-03-09T10:57:17.256 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff98000ac10 2021-03-09T10:57:17.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:17.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:17.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:17.257 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:17.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.247+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:17.258 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.251+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:17.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.251+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:17.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.251+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:17.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.251+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:17.259 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.251+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008ea30 con 0x7ff98000ac10 2021-03-09T10:57:17.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.251+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800267a0 con 0x7ff98000ac10 2021-03-09T10:57:17.260 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:17.255+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34202 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:17.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.255+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 secure :-1 s=READY pgs=10054 cs=0 l=1 rev1=1 rx=0x7ff98c0a9c60 tx=0x7ff98c006650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:17.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.255+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a9c90 con 0x7ff98000ac10 2021-03-09T10:57:17.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.263+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0a9c90 con 0x7ff98000ac10 2021-03-09T10:57:17.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.263+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c052130 con 0x7ff98000ac10 2021-03-09T10:57:17.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.263+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:17.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.263+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:17.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:17.263+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:22.260 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:57:22.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:22.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 secure :-1 s=READY pgs=10054 cs=0 l=1 rev1=1 rx=0x7ff98c0a9c60 tx=0x7ff98c006650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:22.261 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 secure :-1 s=READY pgs=10054 cs=0 l=1 rev1=1 rx=0x7ff98c0a9c60 tx=0x7ff98c006650).stop 2021-03-09T10:57:22.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:22.262 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 unknown :-1 s=CLOSED pgs=10054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:22.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:22.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:22.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:22.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004ba60 con 0x7ff99c0adc20 2021-03-09T10:57:22.263 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006430 con 0x7ff99c0613a0 2021-03-09T10:57:22.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980025f90 con 0x7ff980083100 2021-03-09T10:57:22.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:22.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:22.264 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:22.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:22.265 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.255+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:22.266 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:22.259+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34241 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:22.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.259+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:22.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.259+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:22.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.259+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:22.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.259+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:22.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.259+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800047f0 con 0x7ff980083100 2021-03-09T10:57:22.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.259+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ea30 con 0x7ff980083100 2021-03-09T10:57:22.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.263+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 secure :-1 s=READY pgs=10061 cs=0 l=1 rev1=1 rx=0x7ff980015300 tx=0x7ff98c07ed10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:22.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.267+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00b400 con 0x7ff980083100 2021-03-09T10:57:22.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.267+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c079950 con 0x7ff980083100 2021-03-09T10:57:22.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.267+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c079ac0 con 0x7ff980083100 2021-03-09T10:57:22.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.267+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:22.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.267+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:22.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:22.267+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:27.267 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:27.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:27.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 secure :-1 s=READY pgs=10061 cs=0 l=1 rev1=1 rx=0x7ff980015300 tx=0x7ff98c07ed10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:27.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 secure :-1 s=READY pgs=10061 cs=0 l=1 rev1=1 rx=0x7ff980015300 tx=0x7ff98c07ed10).stop 2021-03-09T10:57:27.268 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:27.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015a80 unknown :-1 s=CLOSED pgs=10061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:27.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:27.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:27.269 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:27.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a870 con 0x7ff99c0adc20 2021-03-09T10:57:27.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004bc00 con 0x7ff99c0613a0 2021-03-09T10:57:27.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980006430 con 0x7ff98000ac10 2021-03-09T10:57:27.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:27.270 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:27.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:27.271 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:27.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:27.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:27.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:27.272 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:27.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:27.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980096980 con 0x7ff98000ac10 2021-03-09T10:57:27.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.263+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800158f0 con 0x7ff98000ac10 2021-03-09T10:57:27.273 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.267+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 secure :-1 s=READY pgs=10069 cs=0 l=1 rev1=1 rx=0x7ff994039040 tx=0x7ff994011470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:27.274 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:27.267+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34277 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:27.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.267+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940a3d00 con 0x7ff98000ac10 2021-03-09T10:57:27.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.267+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99409b190 con 0x7ff98000ac10 2021-03-09T10:57:27.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.275+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:57:27.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.275+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:27.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.275+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:27.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:27.275+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:32.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:32.274 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:32.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 secure :-1 s=READY pgs=10069 cs=0 l=1 rev1=1 rx=0x7ff994039040 tx=0x7ff994011470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:32.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 secure :-1 s=READY pgs=10069 cs=0 l=1 rev1=1 rx=0x7ff994039040 tx=0x7ff994011470).stop 2021-03-09T10:57:32.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980096490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:32.275 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980096490 unknown :-1 s=CLOSED pgs=10069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:32.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:32.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:32.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:32.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a870 con 0x7ff99c0adc20 2021-03-09T10:57:32.276 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ebf0 con 0x7ff99c0613a0 2021-03-09T10:57:32.277 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015c40 con 0x7ff980083100 2021-03-09T10:57:32.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:32.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:32.278 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:32.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:32.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.267+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:32.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.271+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:32.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.271+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:32.279 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.271+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:32.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.271+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:32.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.271+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980015d80 con 0x7ff980083100 2021-03-09T10:57:32.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.271+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980096980 con 0x7ff980083100 2021-03-09T10:57:32.280 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.271+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015750 secure :-1 s=READY pgs=10079 cs=0 l=1 rev1=1 rx=0x7ff98004a020 tx=0x7ff994020fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:32.281 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:32.275+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34316 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:32.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.275+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980083100 2021-03-09T10:57:32.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.275+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980083100 2021-03-09T10:57:32.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.279+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980083100 2021-03-09T10:57:32.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.279+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:32.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.279+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:32.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:32.279+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:36.707 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:57:36.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:57:37.281 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:37.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:37.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015750 secure :-1 s=READY pgs=10079 cs=0 l=1 rev1=1 rx=0x7ff98004a020 tx=0x7ff994020fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:37.282 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015750 secure :-1 s=READY pgs=10079 cs=0 l=1 rev1=1 rx=0x7ff98004a020 tx=0x7ff994020fa0).stop 2021-03-09T10:57:37.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:37.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015750 unknown :-1 s=CLOSED pgs=10079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:37.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:37.283 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:37.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:37.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015c40 con 0x7ff99c0adc20 2021-03-09T10:57:37.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a870 con 0x7ff99c0613a0 2021-03-09T10:57:37.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980056de0 con 0x7ff98000ac10 2021-03-09T10:57:37.284 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:37.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:37.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:37.285 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:37.286 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:37.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:37.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:37.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:37.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:37.287 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980005c30 con 0x7ff98000ac10 2021-03-09T10:57:37.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800047f0 con 0x7ff98000ac10 2021-03-09T10:57:37.288 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.279+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10089 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c006730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:37.288 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:37.283+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34352 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:37.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.283+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04c370 con 0x7ff98000ac10 2021-03-09T10:57:37.289 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.283+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08f320 con 0x7ff98000ac10 2021-03-09T10:57:37.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.287+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:57:37.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.287+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:37.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.287+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:37.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:37.287+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:42.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:42.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:42.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10089 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c006730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:42.290 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10089 cs=0 l=1 rev1=1 rx=0x7ff98c003fd0 tx=0x7ff98c006730).stop 2021-03-09T10:57:42.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:42.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=CLOSED pgs=10089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:42.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:42.291 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800156f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:42.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:42.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000a870 con 0x7ff99c0adc20 2021-03-09T10:57:42.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ebd0 con 0x7ff99c0613a0 2021-03-09T10:57:42.292 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015be0 con 0x7ff980083100 2021-03-09T10:57:42.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:42.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:42.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:42.293 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:42.294 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.283+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800156f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:42.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.287+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:42.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.287+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:42.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.287+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:42.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.287+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:42.295 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.287+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980024d70 con 0x7ff980083100 2021-03-09T10:57:42.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.287+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980005c30 con 0x7ff980083100 2021-03-09T10:57:42.296 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.287+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800156f0 secure :-1 s=READY pgs=10101 cs=0 l=1 rev1=1 rx=0x7ff98c07deb0 tx=0x7ff98c04bfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:42.297 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:42.291+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34391 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:42.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0590d0 con 0x7ff980083100 2021-03-09T10:57:42.297 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0590d0 con 0x7ff980083100 2021-03-09T10:57:42.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.295+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0592f0 con 0x7ff980083100 2021-03-09T10:57:42.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.295+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:42.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.295+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:42.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:42.295+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:47.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800156f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:57:47.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800156f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:47.298 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800156f0 secure :-1 s=READY pgs=10101 cs=0 l=1 rev1=1 rx=0x7ff98c07deb0 tx=0x7ff98c04bfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:47.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800156f0 secure :-1 s=READY pgs=10101 cs=0 l=1 rev1=1 rx=0x7ff98c07deb0 tx=0x7ff98c04bfc0).stop 2021-03-09T10:57:47.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff9800156f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:47.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff9800156f0 unknown :-1 s=CLOSED pgs=10101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:47.299 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:47.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:47.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:47.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000eb20 con 0x7ff99c0adc20 2021-03-09T10:57:47.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d330 con 0x7ff99c0613a0 2021-03-09T10:57:47.300 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:47.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005be0 con 0x7ff98000ac10 2021-03-09T10:57:47.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:47.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T10:57:47.301 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:47.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-03-09T10:57:47.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 msgr2=0x7ff980021bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:47.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:47.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff99c0613a0 0x7ff980021bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:47.302 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 msgr2=0x7ff980039e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:47.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff99c0adc20 0x7ff980039e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:47.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff978009ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:47.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff978005810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:47.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff9780061a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:47.303 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d330 con 0x7ff978005d20 2021-03-09T10:57:47.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005be0 con 0x7ff9780033c0 2021-03-09T10:57:47.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff9927fc700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff978006800 con 0x7ff98000ac10 2021-03-09T10:57:47.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff978005810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:47.304 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff978005810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:47.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff9780061a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:47.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff9780061a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:47.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.291+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff978009ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:47.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.295+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff978005810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:47.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.295+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff978005810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:47.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.295+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff9780061a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:47.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.295+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff9780061a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:47.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.295+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008ea30 con 0x7ff98000ac10 2021-03-09T10:57:47.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.295+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980015d80 con 0x7ff98000ac10 2021-03-09T10:57:47.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.295+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff978009ef0 secure :-1 s=READY pgs=10113 cs=0 l=1 rev1=1 rx=0x7ff980092c30 tx=0x7ff99400bf10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:47.308 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:47.299+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34427 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:47.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff98000ac10 2021-03-09T10:57:47.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994014ea0 con 0x7ff98000ac10 2021-03-09T10:57:47.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.303+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:57:47.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.303+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:47.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.303+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:47.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:47.303+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:52.305 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff978009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:52.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff978009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:52.306 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff978009ef0 secure :-1 s=READY pgs=10113 cs=0 l=1 rev1=1 rx=0x7ff980092c30 tx=0x7ff99400bf10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:52.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff978009ef0 secure :-1 s=READY pgs=10113 cs=0 l=1 rev1=1 rx=0x7ff980092c30 tx=0x7ff99400bf10).stop 2021-03-09T10:57:52.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff978009ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:52.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff978009ef0 unknown :-1 s=CLOSED pgs=10113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:52.307 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:52.308 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:52.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff980024f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:52.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009740 con 0x7ff978005d20 2021-03-09T10:57:52.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000d350 con 0x7ff9780033c0 2021-03-09T10:57:52.309 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020f30 con 0x7ff980083100 2021-03-09T10:57:52.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:52.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:52.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff980024f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:52.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff980024f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:52.310 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.299+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:52.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff980024f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:52.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff980024f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:52.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:52.311 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:52.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800359e0 con 0x7ff980083100 2021-03-09T10:57:52.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.303+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ea30 con 0x7ff980083100 2021-03-09T10:57:52.312 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.303+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015660 secure :-1 s=READY pgs=10125 cs=0 l=1 rev1=1 rx=0x7ff9940aaa70 tx=0x7ff9940aae90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:52.313 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:52.307+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34466 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:52.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994085e10 con 0x7ff980083100 2021-03-09T10:57:52.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.311+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994085e10 con 0x7ff980083100 2021-03-09T10:57:52.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:57:52.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.315+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:52.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.315+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:52.321 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:52.315+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:57:57.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:57.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:57.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015660 secure :-1 s=READY pgs=10125 cs=0 l=1 rev1=1 rx=0x7ff9940aaa70 tx=0x7ff9940aae90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:57.313 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015660 secure :-1 s=READY pgs=10125 cs=0 l=1 rev1=1 rx=0x7ff9940aaa70 tx=0x7ff9940aae90).stop 2021-03-09T10:57:57.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980015660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:57.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980015660 unknown :-1 s=CLOSED pgs=10125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:57.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:57.315 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:57.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:57.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015ee0 con 0x7ff978005d20 2021-03-09T10:57:57.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f1d0 con 0x7ff9780033c0 2021-03-09T10:57:57.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009740 con 0x7ff98000ac10 2021-03-09T10:57:57.316 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:57:57.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:57.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:57:57.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.307+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:57.317 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.311+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:57.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.311+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:57.318 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.311+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:57.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.311+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:57.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.311+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:57.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.311+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000b6e0 con 0x7ff98000ac10 2021-03-09T10:57:57.319 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.311+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008ea30 con 0x7ff98000ac10 2021-03-09T10:57:57.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.311+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10136 cs=0 l=1 rev1=1 rx=0x7ff99409b1d0 tx=0x7ff9940340b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:57.320 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:57:57.315+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34502 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:57:57.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b76b0 con 0x7ff98000ac10 2021-03-09T10:57:57.323 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.319+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940b76b0 con 0x7ff98000ac10 2021-03-09T10:57:57.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:57:57.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.323+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:57.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.323+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:57.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:57:57.323+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:02.320 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:02.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:02.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10136 cs=0 l=1 rev1=1 rx=0x7ff99409b1d0 tx=0x7ff9940340b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:02.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10136 cs=0 l=1 rev1=1 rx=0x7ff99409b1d0 tx=0x7ff9940340b0).stop 2021-03-09T10:58:02.440 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:02.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=CLOSED pgs=10136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:02.441 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:02.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:02.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:02.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800498e0 con 0x7ff978005d20 2021-03-09T10:58:02.443 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980005d30 con 0x7ff9780033c0 2021-03-09T10:58:02.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980015310 con 0x7ff980083100 2021-03-09T10:58:02.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:02.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:02.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:02.444 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:02.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.315+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:02.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.319+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:02.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.319+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:02.445 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.319+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:02.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.319+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:02.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.319+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800210b0 con 0x7ff980083100 2021-03-09T10:58:02.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.319+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980024d70 con 0x7ff980083100 2021-03-09T10:58:02.446 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.319+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024f00 secure :-1 s=READY pgs=10144 cs=0 l=1 rev1=1 rx=0x7ff98c07dc30 tx=0x7ff98c00bef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:02.447 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:02.319+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34541 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:02.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980083100 2021-03-09T10:58:02.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff980083100 2021-03-09T10:58:02.448 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:58:02.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.327+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:02.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.327+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:02.449 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:02.327+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:06.772 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:58:06.776 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:58:07.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980024f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:07.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980024f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:07.327 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024f00 secure :-1 s=READY pgs=10144 cs=0 l=1 rev1=1 rx=0x7ff98c07dc30 tx=0x7ff98c00bef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:07.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024f00 secure :-1 s=READY pgs=10144 cs=0 l=1 rev1=1 rx=0x7ff98c07dc30 tx=0x7ff98c00bef0).stop 2021-03-09T10:58:07.328 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980024f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:07.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980024f00 unknown :-1 s=CLOSED pgs=10144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:07.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:07.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:07.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:07.329 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800292b0 con 0x7ff978005d20 2021-03-09T10:58:07.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f1d0 con 0x7ff9780033c0 2021-03-09T10:58:07.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b640 con 0x7ff98000ac10 2021-03-09T10:58:07.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:07.330 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:07.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:07.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:07.331 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:07.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:07.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:07.332 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:07.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:07.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980024d70 con 0x7ff98000ac10 2021-03-09T10:58:07.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000b6e0 con 0x7ff98000ac10 2021-03-09T10:58:07.333 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.323+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10153 cs=0 l=1 rev1=1 rx=0x7ff994083280 tx=0x7ff9940702d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:07.334 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:07.327+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34577 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:07.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994020ea0 con 0x7ff98000ac10 2021-03-09T10:58:07.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400d1c0 con 0x7ff98000ac10 2021-03-09T10:58:07.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:58:07.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.335+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:07.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.335+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:07.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:07.335+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:12.335 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.327+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:12.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.327+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:12.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10153 cs=0 l=1 rev1=1 rx=0x7ff994083280 tx=0x7ff9940702d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:12.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.327+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10153 cs=0 l=1 rev1=1 rx=0x7ff994083280 tx=0x7ff9940702d0).stop 2021-03-09T10:58:12.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.327+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:12.336 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=CLOSED pgs=10153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:12.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:12.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:12.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.327+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:12.337 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ebd0 con 0x7ff978005d20 2021-03-09T10:58:12.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800292b0 con 0x7ff9780033c0 2021-03-09T10:58:12.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98002d780 con 0x7ff980083100 2021-03-09T10:58:12.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:12.338 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:12.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:12.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:12.339 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:12.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:12.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:12.340 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:12.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:12.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800359e0 con 0x7ff980083100 2021-03-09T10:58:12.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.331+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800210b0 con 0x7ff980083100 2021-03-09T10:58:12.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.335+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008faf0 secure :-1 s=READY pgs=10167 cs=0 l=1 rev1=1 rx=0x7ff98c07a280 tx=0x7ff98c07d300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:12.342 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:12.335+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34617 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:12.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.335+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:58:12.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.339+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07d0f0 con 0x7ff980083100 2021-03-09T10:58:12.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.343+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:58:12.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.343+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:12.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.343+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:12.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:12.343+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:17.341 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:17.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:17.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008faf0 secure :-1 s=READY pgs=10167 cs=0 l=1 rev1=1 rx=0x7ff98c07a280 tx=0x7ff98c07d300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:17.342 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008faf0 secure :-1 s=READY pgs=10167 cs=0 l=1 rev1=1 rx=0x7ff98c07a280 tx=0x7ff98c07d300).stop 2021-03-09T10:58:17.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008faf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:17.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008faf0 unknown :-1 s=CLOSED pgs=10167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:17.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:17.343 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:17.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:17.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800292b0 con 0x7ff978005d20 2021-03-09T10:58:17.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980020f30 con 0x7ff9780033c0 2021-03-09T10:58:17.344 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022120 con 0x7ff98000ac10 2021-03-09T10:58:17.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:17.345 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:17.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:17.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:17.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:17.346 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:17.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:17.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:17.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:17.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000b6e0 con 0x7ff98000ac10 2021-03-09T10:58:17.347 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800359e0 con 0x7ff98000ac10 2021-03-09T10:58:17.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.338+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10175 cs=0 l=1 rev1=1 rx=0x7ff980024700 tx=0x7ff98c005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:17.349 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:17.342+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34652 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:17.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00b120 con 0x7ff98000ac10 2021-03-09T10:58:17.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00b280 con 0x7ff98000ac10 2021-03-09T10:58:17.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.346+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c00b120 con 0x7ff98000ac10 2021-03-09T10:58:17.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.346+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:17.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.346+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:17.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:17.346+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:22.348 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:22.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:22.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10175 cs=0 l=1 rev1=1 rx=0x7ff980024700 tx=0x7ff98c005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:22.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 secure :-1 s=READY pgs=10175 cs=0 l=1 rev1=1 rx=0x7ff980024700 tx=0x7ff98c005110).stop 2021-03-09T10:58:22.349 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980024880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:22.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980024880 unknown :-1 s=CLOSED pgs=10175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:22.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:22.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:22.350 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:22.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ebd0 con 0x7ff978005d20 2021-03-09T10:58:22.351 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980057630 con 0x7ff9780033c0 2021-03-09T10:58:22.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f1d0 con 0x7ff980083100 2021-03-09T10:58:22.352 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:22.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:22.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:22.353 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:22.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:22.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:22.354 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:22.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:22.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:22.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980049a60 con 0x7ff980083100 2021-03-09T10:58:22.355 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.342+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980024d70 con 0x7ff980083100 2021-03-09T10:58:22.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.346+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008fa80 secure :-1 s=READY pgs=10185 cs=0 l=1 rev1=1 rx=0x7ff99400d380 tx=0x7ff994034c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:22.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994098120 con 0x7ff980083100 2021-03-09T10:58:22.357 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:22.346+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34692 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:22.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994098350 con 0x7ff980083100 2021-03-09T10:58:22.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.354+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:58:22.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.354+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:22.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.354+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:22.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:22.354+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:27.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:27.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:27.356 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008fa80 secure :-1 s=READY pgs=10185 cs=0 l=1 rev1=1 rx=0x7ff99400d380 tx=0x7ff994034c50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:27.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008fa80 secure :-1 s=READY pgs=10185 cs=0 l=1 rev1=1 rx=0x7ff99400d380 tx=0x7ff994034c50).stop 2021-03-09T10:58:27.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008fa80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:27.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008fa80 unknown :-1 s=CLOSED pgs=10185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:27.357 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:27.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:27.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:27.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008ebd0 con 0x7ff978005d20 2021-03-09T10:58:27.358 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000f1d0 con 0x7ff9780033c0 2021-03-09T10:58:27.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b640 con 0x7ff98000ac10 2021-03-09T10:58:27.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:27.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:27.359 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:27.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:27.360 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.350+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:27.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:27.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:27.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:27.361 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:27.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980057630 con 0x7ff98000ac10 2021-03-09T10:58:27.362 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.354+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000b6e0 con 0x7ff98000ac10 2021-03-09T10:58:27.363 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:27.354+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34727 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:27.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.354+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 secure :-1 s=READY pgs=10202 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c00baf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:27.363 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c006650 con 0x7ff98000ac10 2021-03-09T10:58:27.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c07deb0 con 0x7ff98000ac10 2021-03-09T10:58:27.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff98000ac10 2021-03-09T10:58:27.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.362+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:27.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.362+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:27.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:27.362+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:32.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:32.364 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:32.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 secure :-1 s=READY pgs=10202 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c00baf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:32.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 secure :-1 s=READY pgs=10202 cs=0 l=1 rev1=1 rx=0x7ff98c00e040 tx=0x7ff98c00baf0).stop 2021-03-09T10:58:32.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:32.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 unknown :-1 s=CLOSED pgs=10202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:32.365 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:32.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:32.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:32.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b640 con 0x7ff978005d20 2021-03-09T10:58:32.366 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b6e0 con 0x7ff9780033c0 2021-03-09T10:58:32.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800359e0 con 0x7ff980083100 2021-03-09T10:58:32.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:32.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:32.367 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:32.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:32.368 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:32.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:32.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:32.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:32.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:32.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98008fd80 con 0x7ff980083100 2021-03-09T10:58:32.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980049a60 con 0x7ff980083100 2021-03-09T10:58:32.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.358+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f890 secure :-1 s=READY pgs=10207 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994098190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:32.371 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:32.362+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34765 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:32.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980083100 2021-03-09T10:58:32.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994081670 con 0x7ff980083100 2021-03-09T10:58:32.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.366+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff980083100 2021-03-09T10:58:32.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.366+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:32.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.370+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:32.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:32.370+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:36.929 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:58:36.932 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:58:37.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:37.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:37.369 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f890 secure :-1 s=READY pgs=10207 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994098190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:37.370 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f890 secure :-1 s=READY pgs=10207 cs=0 l=1 rev1=1 rx=0x7ff994009460 tx=0x7ff994098190).stop 2021-03-09T10:58:37.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008f890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:37.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f890 unknown :-1 s=CLOSED pgs=10207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:37.371 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:37.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:37.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:37.372 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:37.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b640 con 0x7ff978005d20 2021-03-09T10:58:37.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:37.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b6e0 con 0x7ff9780033c0 2021-03-09T10:58:37.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800261d0 con 0x7ff98000ac10 2021-03-09T10:58:37.373 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:37.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.362+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:37.374 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:37.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:37.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:37.375 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:37.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.366+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:37.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.366+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980026340 con 0x7ff98000ac10 2021-03-09T10:58:37.376 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.370+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980057630 con 0x7ff98000ac10 2021-03-09T10:58:37.377 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:37.370+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34802 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:37.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.370+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 secure :-1 s=READY pgs=10224 cs=0 l=1 rev1=1 rx=0x7ff98003a410 tx=0x7ff98c07deb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:37.377 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08d6c0 con 0x7ff98000ac10 2021-03-09T10:58:37.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08d070 con 0x7ff98000ac10 2021-03-09T10:58:37.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0a2aa0 con 0x7ff98000ac10 2021-03-09T10:58:37.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.378+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:37.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.378+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:37.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:37.378+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:42.378 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:42.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:42.379 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 secure :-1 s=READY pgs=10224 cs=0 l=1 rev1=1 rx=0x7ff98003a410 tx=0x7ff98c07deb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:42.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 secure :-1 s=READY pgs=10224 cs=0 l=1 rev1=1 rx=0x7ff98003a410 tx=0x7ff98c07deb0).stop 2021-03-09T10:58:42.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:42.380 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 unknown :-1 s=CLOSED pgs=10224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:42.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:42.381 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:42.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:42.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022e50 con 0x7ff978005d20 2021-03-09T10:58:42.382 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980055030 con 0x7ff9780033c0 2021-03-09T10:58:42.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:42.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800261d0 con 0x7ff980083100 2021-03-09T10:58:42.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:42.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:42.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:42.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.370+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:42.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:42.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:42.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:42.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:42.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980057630 con 0x7ff980083100 2021-03-09T10:58:42.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.374+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980026340 con 0x7ff980083100 2021-03-09T10:58:42.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.374+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f550 secure :-1 s=READY pgs=10230 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c08f130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:42.389 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:42.374+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34840 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:42.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c08d6c0 con 0x7ff980083100 2021-03-09T10:58:42.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c08d6c0 con 0x7ff980083100 2021-03-09T10:58:42.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.382+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c08dd00 con 0x7ff980083100 2021-03-09T10:58:42.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.382+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:42.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.382+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:42.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:42.382+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:47.383 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:47.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:47.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f550 secure :-1 s=READY pgs=10230 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c08f130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:47.384 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f550 secure :-1 s=READY pgs=10230 cs=0 l=1 rev1=1 rx=0x7ff98c006b30 tx=0x7ff98c08f130).stop 2021-03-09T10:58:47.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008f550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:47.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f550 unknown :-1 s=CLOSED pgs=10230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:47.385 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:47.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:47.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:47.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:47.386 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980022e30 con 0x7ff978005d20 2021-03-09T10:58:47.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:47.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800261d0 con 0x7ff9780033c0 2021-03-09T10:58:47.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026310 con 0x7ff98000ac10 2021-03-09T10:58:47.387 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:47.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:47.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.378+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:47.388 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.382+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:47.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.382+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:47.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.382+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:47.389 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.382+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:47.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.382+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff98000b6e0 con 0x7ff98000ac10 2021-03-09T10:58:47.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.382+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98008fd80 con 0x7ff98000ac10 2021-03-09T10:58:47.390 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.382+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 secure :-1 s=READY pgs=10244 cs=0 l=1 rev1=1 rx=0x7ff99400a720 tx=0x7ff994029f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:47.391 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:47.382+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34877 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:47.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400cd10 con 0x7ff98000ac10 2021-03-09T10:58:47.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400a750 con 0x7ff98000ac10 2021-03-09T10:58:47.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:58:47.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.390+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:47.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.390+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:47.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:47.390+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:52.391 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:52.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:52.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 secure :-1 s=READY pgs=10244 cs=0 l=1 rev1=1 rx=0x7ff99400a720 tx=0x7ff994029f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:52.392 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 secure :-1 s=READY pgs=10244 cs=0 l=1 rev1=1 rx=0x7ff99400a720 tx=0x7ff994029f10).stop 2021-03-09T10:58:52.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff980017310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:52.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff980017310 unknown :-1 s=CLOSED pgs=10244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:52.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:52.393 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:52.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:52.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:52.394 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:52.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026370 con 0x7ff978005d20 2021-03-09T10:58:52.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009980 con 0x7ff9780033c0 2021-03-09T10:58:52.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800359e0 con 0x7ff980083100 2021-03-09T10:58:52.395 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:52.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:52.396 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:52.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:52.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:52.397 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:52.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:52.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff980022de0 con 0x7ff980083100 2021-03-09T10:58:52.398 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff980057630 con 0x7ff980083100 2021-03-09T10:58:52.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.386+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f4b0 secure :-1 s=READY pgs=10249 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c02d010).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:52.399 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:52.390+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34914 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:52.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00c150 con 0x7ff980083100 2021-03-09T10:58:52.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.390+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c005110 con 0x7ff980083100 2021-03-09T10:58:52.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.398+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff98c0871d0 con 0x7ff980083100 2021-03-09T10:58:52.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.398+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:52.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.398+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:52.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:52.398+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:58:57.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:57.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:57.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f4b0 secure :-1 s=READY pgs=10249 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c02d010).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:57.399 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f4b0 secure :-1 s=READY pgs=10249 cs=0 l=1 rev1=1 rx=0x7ff98c07b060 tx=0x7ff98c02d010).stop 2021-03-09T10:58:57.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff98008f4b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:57.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff98008f4b0 unknown :-1 s=CLOSED pgs=10249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:57.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:57.400 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:57.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:57.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980026370 con 0x7ff978005d20 2021-03-09T10:58:57.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009980 con 0x7ff9780033c0 2021-03-09T10:58:57.401 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800172a0 con 0x7ff98000ac10 2021-03-09T10:58:57.402 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:58:57.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:58:57.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:57.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:57.403 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:57.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:57.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:57.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:57.404 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:57.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800173e0 con 0x7ff98000ac10 2021-03-09T10:58:57.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.394+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98000b6e0 con 0x7ff98000ac10 2021-03-09T10:58:57.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.398+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 secure :-1 s=READY pgs=10264 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994085f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:57.406 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:58:57.398+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34951 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:58:57.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.398+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994068130 con 0x7ff98000ac10 2021-03-09T10:58:57.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.406+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994020220 con 0x7ff98000ac10 2021-03-09T10:58:57.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.406+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994009950 con 0x7ff98000ac10 2021-03-09T10:58:57.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.406+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:57.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.406+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:57.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:58:57.406+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:59:02.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:59:02.405 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:02.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 secure :-1 s=READY pgs=10264 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994085f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:02.406 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 secure :-1 s=READY pgs=10264 cs=0 l=1 rev1=1 rx=0x7ff99400b900 tx=0x7ff994085f70).stop 2021-03-09T10:59:02.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008fb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:02.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 unknown :-1 s=CLOSED pgs=10264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:02.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980017570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:02.407 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:02.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:02.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800172a0 con 0x7ff978005d20 2021-03-09T10:59:02.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980009980 con 0x7ff9780033c0 2021-03-09T10:59:02.408 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98004b860 con 0x7ff980083100 2021-03-09T10:59:02.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:59:02.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:02.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:59:02.409 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:02.410 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980017570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:02.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:02.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:02.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:02.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:02.411 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800359e0 con 0x7ff980083100 2021-03-09T10:59:02.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800173e0 con 0x7ff980083100 2021-03-09T10:59:02.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.402+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980017570 secure :-1 s=READY pgs=10273 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99405f480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:02.412 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:59:02.406+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34989 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:59:02.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.406+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980083100 2021-03-09T10:59:02.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.410+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff980083100 2021-03-09T10:59:02.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.414+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980083100 2021-03-09T10:59:02.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.414+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:59:02.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.414+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:59:02.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:02.414+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:59:07.222 DEBUG:teuthology.orchestra.run.smithi052:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:59:07.225 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:59:07.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980017570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:59:07.412 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980017570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:07.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980017570 secure :-1 s=READY pgs=10273 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99405f480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:07.413 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980017570 secure :-1 s=READY pgs=10273 cs=0 l=1 rev1=1 rx=0x7ff99400ba90 tx=0x7ff99405f480).stop 2021-03-09T10:59:07.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 msgr2=0x7ff980017570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:07.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980017570 unknown :-1 s=CLOSED pgs=10273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:07.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:07.414 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:07.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:07.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800172a0 con 0x7ff978005d20 2021-03-09T10:59:07.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d3f0 con 0x7ff9780033c0 2021-03-09T10:59:07.415 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980016240 con 0x7ff98000ac10 2021-03-09T10:59:07.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:59:07.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:07.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:59:07.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:07.416 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:07.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:07.417 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:07.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:07.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:07.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800359e0 con 0x7ff98000ac10 2021-03-09T10:59:07.418 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800173e0 con 0x7ff98000ac10 2021-03-09T10:59:07.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.410+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 secure :-1 s=READY pgs=10283 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff9940ace80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:07.419 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:59:07.410+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35026 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:59:07.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.414+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff98000ac10 2021-03-09T10:59:07.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.418+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408d180 con 0x7ff98000ac10 2021-03-09T10:59:07.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.422+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff99408e5e0 con 0x7ff98000ac10 2021-03-09T10:59:07.427 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.422+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:59:07.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.422+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:59:07.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:07.422+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:59:12.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:59:12.419 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:12.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 secure :-1 s=READY pgs=10283 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff9940ace80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:12.420 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 secure :-1 s=READY pgs=10283 cs=0 l=1 rev1=1 rx=0x7ff99401cb60 tx=0x7ff9940ace80).stop 2021-03-09T10:59:12.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 msgr2=0x7ff98008fb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:12.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff98000ac10 0x7ff98008fb40 unknown :-1 s=CLOSED pgs=10283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:12.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:12.421 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:12.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff9937fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980017570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:12.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800172a0 con 0x7ff978005d20 2021-03-09T10:59:12.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800901d0 con 0x7ff9780033c0 2021-03-09T10:59:12.422 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98008d3f0 con 0x7ff980083100 2021-03-09T10:59:12.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff99bfff700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:3300/0 2021-03-09T10:59:12.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff99bfff700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:12.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:3300/0 2021-03-09T10:59:12.423 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:12.424 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.414+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980017570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:12.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.418+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 msgr2=0x7ff98000cfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:12.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.418+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff9780033c0 0x7ff98000cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:12.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.418+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 msgr2=0x7ff980021a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:12.425 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.418+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff978005d20 0x7ff980021a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:12.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.418+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff9800359e0 con 0x7ff980083100 2021-03-09T10:59:12.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.418+0000 7ff99b7fe700 1 -- 172.21.15.52:0/1490464907 --> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9800173e0 con 0x7ff980083100 2021-03-09T10:59:12.426 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.418+0000 7ff99b7fe700 1 --2- 172.21.15.52:0/1490464907 >> [v2:172.21.15.52:3301/0,v1:172.21.15.52:6790/0] conn(0x7ff980083100 0x7ff980017570 secure :-1 s=READY pgs=10297 cs=0 l=1 rev1=1 rx=0x7ff99405f560 tx=0x7ff994020f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:12.427 INFO:tasks.ceph.mon.c.smithi052.stderr:2021-03-09T10:59:12.418+0000 7f0857402700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35062 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:38:29.148614+0000) 2021-03-09T10:59:12.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.418+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99408e420 con 0x7ff980083100 2021-03-09T10:59:12.428 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.422+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99408e240 con 0x7ff980083100 2021-03-09T10:59:12.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.426+0000 7ff9937fe700 1 -- 172.21.15.52:0/1490464907 <== mon.2 v2:172.21.15.52:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7ff994001140 con 0x7ff980083100 2021-03-09T10:59:12.431 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.426+0000 7ff9937fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:59:12.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.426+0000 7ff9937fe700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:59:12.432 INFO:teuthology.misc.health.smithi052.stderr:2021-03-09T10:59:12.426+0000 7ff9937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.52:6808/14433,v1:172.21.15.52:6809/14433] 2021-03-09T10:59:15.597 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-03-09T10:59:15.598 DEBUG:teuthology.task.console_log:Killing console logger for smithi052 2021-03-09T10:59:15.599 DEBUG:teuthology.task.console_log:Killing console logger for smithi194 2021-03-09T10:59:15.599 DEBUG:teuthology.exit:Finished running handlers